mirrors/cosmopolitan
1
0
Fork 0
mirror of https://github.com/jart/cosmopolitan.git synced 2025-05-31 17:52:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

python-3.6.zip added from Github

Browse source 
README.cosmo contains the necessary links.
This commit is contained in:
ahgamut 2021-08-08 09:38:33 +05:30 • committed by Justine Tunney
parent 75fc601ff5
commit 0c4c56ff39
4219 changed files with 1968626 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

474
third_party/python/Doc/library/2to3.rst vendored Normal file
View file

@ -0,0 +1,474 @@
.. _2to3-reference:
2to3 - Automated Python 2 to 3 code translation
===============================================
.. sectionauthor:: Benjamin Peterson <benjamin@python.org>
2to3 is a Python program that reads Python 2.x source code and applies a series
of *fixers* to transform it into valid Python 3.x code. The standard library
contains a rich set of fixers that will handle almost all code. 2to3 supporting
library :mod:`lib2to3` is, however, a flexible and generic library, so it is
possible to write your own fixers for 2to3. :mod:`lib2to3` could also be
adapted to custom applications in which Python code needs to be edited
automatically.
.. _2to3-using:
Using 2to3
----------
2to3 will usually be installed with the Python interpreter as a script. It is
also located in the :file:`Tools/scripts` directory of the Python root.
2to3's basic arguments are a list of files or directories to transform. The
directories are recursively traversed for Python sources.
Here is a sample Python 2.x source file, :file:`example.py`::
def greet(name):
print "Hello, {0}!".format(name)
print "What's your name?"
name = raw_input()
greet(name)
It can be converted to Python 3.x code via 2to3 on the command line:
.. code-block:: shell-session
$ 2to3 example.py
A diff against the original source file is printed. 2to3 can also write the
needed modifications right back to the source file. (A backup of the original
file is made unless :option:`!-n` is also given.) Writing the changes back is
enabled with the :option:`!-w` flag:
.. code-block:: shell-session
$ 2to3 -w example.py
After transformation, :file:`example.py` looks like this::
def greet(name):
print("Hello, {0}!".format(name))
print("What's your name?")
name = input()
greet(name)
Comments and exact indentation are preserved throughout the translation process.
By default, 2to3 runs a set of :ref:`predefined fixers <2to3-fixers>`. The
:option:`!-l` flag lists all available fixers. An explicit set of fixers to run
can be given with :option:`!-f`. Likewise the :option:`!-x` explicitly disables a
fixer. The following example runs only the ``imports`` and ``has_key`` fixers:
.. code-block:: shell-session
$ 2to3 -f imports -f has_key example.py
This command runs every fixer except the ``apply`` fixer:
.. code-block:: shell-session
$ 2to3 -x apply example.py
Some fixers are *explicit*, meaning they aren't run by default and must be
listed on the command line to be run. Here, in addition to the default fixers,
the ``idioms`` fixer is run:
.. code-block:: shell-session
$ 2to3 -f all -f idioms example.py
Notice how passing ``all`` enables all default fixers.
Sometimes 2to3 will find a place in your source code that needs to be changed,
but 2to3 cannot fix automatically. In this case, 2to3 will print a warning
beneath the diff for a file. You should address the warning in order to have
compliant 3.x code.
2to3 can also refactor doctests. To enable this mode, use the :option:`!-d`
flag. Note that *only* doctests will be refactored. This also doesn't require
the module to be valid Python. For example, doctest like examples in a reST
document could also be refactored with this option.
The :option:`!-v` option enables output of more information on the translation
process.
Since some print statements can be parsed as function calls or statements, 2to3
cannot always read files containing the print function. When 2to3 detects the
presence of the ``from __future__ import print_function`` compiler directive, it
modifies its internal grammar to interpret :func:`print` as a function. This
change can also be enabled manually with the :option:`!-p` flag. Use
:option:`!-p` to run fixers on code that already has had its print statements
converted.
The :option:`!-o` or :option:`!--output-dir` option allows specification of an
alternate directory for processed output files to be written to. The
:option:`!-n` flag is required when using this as backup files do not make sense
when not overwriting the input files.
.. versionadded:: 3.2.3
The :option:`!-o` option was added.
The :option:`!-W` or :option:`!--write-unchanged-files` flag tells 2to3 to always
write output files even if no changes were required to the file. This is most
useful with :option:`!-o` so that an entire Python source tree is copied with
translation from one directory to another.
This option implies the :option:`!-w` flag as it would not make sense otherwise.
.. versionadded:: 3.2.3
The :option:`!-W` flag was added.
The :option:`!--add-suffix` option specifies a string to append to all output
filenames. The :option:`!-n` flag is required when specifying this as backups
are not necessary when writing to different filenames. Example:
.. code-block:: shell-session
$ 2to3 -n -W --add-suffix=3 example.py
Will cause a converted file named ``example.py3`` to be written.
.. versionadded:: 3.2.3
The :option:`!--add-suffix` option was added.
To translate an entire project from one directory tree to another use:
.. code-block:: shell-session
$ 2to3 --output-dir=python3-version/mycode -W -n python2-version/mycode
.. _2to3-fixers:
Fixers
------
Each step of transforming code is encapsulated in a fixer. The command ``2to3
-l`` lists them. As :ref:`documented above <2to3-using>`, each can be turned on
and off individually. They are described here in more detail.
.. 2to3fixer:: apply
Removes usage of :func:`apply`. For example ``apply(function, *args,
**kwargs)`` is converted to ``function(*args, **kwargs)``.
.. 2to3fixer:: asserts
Replaces deprecated :mod:`unittest` method names with the correct ones.
================================ ==========================================
From To
================================ ==========================================
``failUnlessEqual(a, b)`` :meth:`assertEqual(a, b)
<unittest.TestCase.assertEqual>`
``assertEquals(a, b)`` :meth:`assertEqual(a, b)
<unittest.TestCase.assertEqual>`
``failIfEqual(a, b)`` :meth:`assertNotEqual(a, b)
<unittest.TestCase.assertNotEqual>`
``assertNotEquals(a, b)`` :meth:`assertNotEqual(a, b)
<unittest.TestCase.assertNotEqual>`
``failUnless(a)`` :meth:`assertTrue(a)
<unittest.TestCase.assertTrue>`
``assert_(a)`` :meth:`assertTrue(a)
<unittest.TestCase.assertTrue>`
``failIf(a)`` :meth:`assertFalse(a)
<unittest.TestCase.assertFalse>`
``failUnlessRaises(exc, cal)`` :meth:`assertRaises(exc, cal)
<unittest.TestCase.assertRaises>`
``failUnlessAlmostEqual(a, b)`` :meth:`assertAlmostEqual(a, b)
<unittest.TestCase.assertAlmostEqual>`
``assertAlmostEquals(a, b)`` :meth:`assertAlmostEqual(a, b)
<unittest.TestCase.assertAlmostEqual>`
``failIfAlmostEqual(a, b)`` :meth:`assertNotAlmostEqual(a, b)
<unittest.TestCase.assertNotAlmostEqual>`
``assertNotAlmostEquals(a, b)`` :meth:`assertNotAlmostEqual(a, b)
<unittest.TestCase.assertNotAlmostEqual>`
================================ ==========================================
.. 2to3fixer:: basestring
Converts :class:`basestring` to :class:`str`.
.. 2to3fixer:: buffer
Converts :class:`buffer` to :class:`memoryview`. This fixer is optional
because the :class:`memoryview` API is similar but not exactly the same as
that of :class:`buffer`.
.. 2to3fixer:: dict
Fixes dictionary iteration methods. :meth:`dict.iteritems` is converted to
:meth:`dict.items`, :meth:`dict.iterkeys` to :meth:`dict.keys`, and
:meth:`dict.itervalues` to :meth:`dict.values`. Similarly,
:meth:`dict.viewitems`, :meth:`dict.viewkeys` and :meth:`dict.viewvalues` are
converted respectively to :meth:`dict.items`, :meth:`dict.keys` and
:meth:`dict.values`. It also wraps existing usages of :meth:`dict.items`,
:meth:`dict.keys`, and :meth:`dict.values` in a call to :class:`list`.
.. 2to3fixer:: except
Converts ``except X, T`` to ``except X as T``.
.. 2to3fixer:: exec
Converts the ``exec`` statement to the :func:`exec` function.
.. 2to3fixer:: execfile
Removes usage of :func:`execfile`. The argument to :func:`execfile` is
wrapped in calls to :func:`open`, :func:`compile`, and :func:`exec`.
.. 2to3fixer:: exitfunc
Changes assignment of :attr:`sys.exitfunc` to use of the :mod:`atexit`
module.
.. 2to3fixer:: filter
Wraps :func:`filter` usage in a :class:`list` call.
.. 2to3fixer:: funcattrs
Fixes function attributes that have been renamed. For example,
``my_function.func_closure`` is converted to ``my_function.__closure__``.
.. 2to3fixer:: future
Removes ``from __future__ import new_feature`` statements.
.. 2to3fixer:: getcwdu
Renames :func:`os.getcwdu` to :func:`os.getcwd`.
.. 2to3fixer:: has_key
Changes ``dict.has_key(key)`` to ``key in dict``.
.. 2to3fixer:: idioms
This optional fixer performs several transformations that make Python code
more idiomatic. Type comparisons like ``type(x) is SomeClass`` and
``type(x) == SomeClass`` are converted to ``isinstance(x, SomeClass)``.
``while 1`` becomes ``while True``. This fixer also tries to make use of
:func:`sorted` in appropriate places. For example, this block ::
L = list(some_iterable)
L.sort()
is changed to ::
L = sorted(some_iterable)
.. 2to3fixer:: import
Detects sibling imports and converts them to relative imports.
.. 2to3fixer:: imports
Handles module renames in the standard library.
.. 2to3fixer:: imports2
Handles other modules renames in the standard library. It is separate from
the :2to3fixer:`imports` fixer only because of technical limitations.
.. 2to3fixer:: input
Converts ``input(prompt)`` to ``eval(input(prompt))``.
.. 2to3fixer:: intern
Converts :func:`intern` to :func:`sys.intern`.
.. 2to3fixer:: isinstance
Fixes duplicate types in the second argument of :func:`isinstance`. For
example, ``isinstance(x, (int, int))`` is converted to ``isinstance(x,
int)`` and ``isinstance(x, (int, float, int))`` is converted to
``isinstance(x, (int, float))``.
.. 2to3fixer:: itertools_imports
Removes imports of :func:`itertools.ifilter`, :func:`itertools.izip`, and
:func:`itertools.imap`. Imports of :func:`itertools.ifilterfalse` are also
changed to :func:`itertools.filterfalse`.
.. 2to3fixer:: itertools
Changes usage of :func:`itertools.ifilter`, :func:`itertools.izip`, and
:func:`itertools.imap` to their built-in equivalents.
:func:`itertools.ifilterfalse` is changed to :func:`itertools.filterfalse`.
.. 2to3fixer:: long
Renames :class:`long` to :class:`int`.
.. 2to3fixer:: map
Wraps :func:`map` in a :class:`list` call. It also changes ``map(None, x)``
to ``list(x)``. Using ``from future_builtins import map`` disables this
fixer.
.. 2to3fixer:: metaclass
Converts the old metaclass syntax (``__metaclass__ = Meta`` in the class
body) to the new (``class X(metaclass=Meta)``).
.. 2to3fixer:: methodattrs
Fixes old method attribute names. For example, ``meth.im_func`` is converted
to ``meth.__func__``.
.. 2to3fixer:: ne
Converts the old not-equal syntax, ``<>``, to ``!=``.
.. 2to3fixer:: next
Converts the use of iterator's :meth:`~iterator.next` methods to the
:func:`next` function. It also renames :meth:`next` methods to
:meth:`~iterator.__next__`.
.. 2to3fixer:: nonzero
Renames :meth:`__nonzero__` to :meth:`~object.__bool__`.
.. 2to3fixer:: numliterals
Converts octal literals into the new syntax.
.. 2to3fixer:: operator
Converts calls to various functions in the :mod:`operator` module to other,
but equivalent, function calls. When needed, the appropriate ``import``
statements are added, e.g. ``import collections``. The following mapping
are made:
================================== ==========================================
From To
================================== ==========================================
``operator.isCallable(obj)`` ``hasattr(obj, '__call__')``
``operator.sequenceIncludes(obj)`` ``operator.contains(obj)``
``operator.isSequenceType(obj)`` ``isinstance(obj, collections.Sequence)``
``operator.isMappingType(obj)`` ``isinstance(obj, collections.Mapping)``
``operator.isNumberType(obj)`` ``isinstance(obj, numbers.Number)``
``operator.repeat(obj, n)`` ``operator.mul(obj, n)``
``operator.irepeat(obj, n)`` ``operator.imul(obj, n)``
================================== ==========================================
.. 2to3fixer:: paren
Add extra parenthesis where they are required in list comprehensions. For
example, ``[x for x in 1, 2]`` becomes ``[x for x in (1, 2)]``.
.. 2to3fixer:: print
Converts the ``print`` statement to the :func:`print` function.
.. 2to3fixer:: raise
Converts ``raise E, V`` to ``raise E(V)``, and ``raise E, V, T`` to ``raise
E(V).with_traceback(T)``. If ``E`` is a tuple, the translation will be
incorrect because substituting tuples for exceptions has been removed in 3.0.
.. 2to3fixer:: raw_input
Converts :func:`raw_input` to :func:`input`.
.. 2to3fixer:: reduce
Handles the move of :func:`reduce` to :func:`functools.reduce`.
.. 2to3fixer:: reload
Converts :func:`reload` to :func:`imp.reload`.
.. 2to3fixer:: renames
Changes :data:`sys.maxint` to :data:`sys.maxsize`.
.. 2to3fixer:: repr
Replaces backtick repr with the :func:`repr` function.
.. 2to3fixer:: set_literal
Replaces use of the :class:`set` constructor with set literals. This fixer
is optional.
.. 2to3fixer:: standarderror
Renames :exc:`StandardError` to :exc:`Exception`.
.. 2to3fixer:: sys_exc
Changes the deprecated :data:`sys.exc_value`, :data:`sys.exc_type`,
:data:`sys.exc_traceback` to use :func:`sys.exc_info`.
.. 2to3fixer:: throw
Fixes the API change in generator's :meth:`throw` method.
.. 2to3fixer:: tuple_params
Removes implicit tuple parameter unpacking. This fixer inserts temporary
variables.
.. 2to3fixer:: types
Fixes code broken from the removal of some members in the :mod:`types`
module.
.. 2to3fixer:: unicode
Renames :class:`unicode` to :class:`str`.
.. 2to3fixer:: urllib
Handles the rename of :mod:`urllib` and :mod:`urllib2` to the :mod:`urllib`
package.
.. 2to3fixer:: ws_comma
Removes excess whitespace from comma separated items. This fixer is
optional.
.. 2to3fixer:: xrange
Renames :func:`xrange` to :func:`range` and wraps existing :func:`range`
calls with :class:`list`.
.. 2to3fixer:: xreadlines
Changes ``for x in file.xreadlines()`` to ``for x in file``.
.. 2to3fixer:: zip
Wraps :func:`zip` usage in a :class:`list` call. This is disabled when
``from future_builtins import zip`` appears.
:mod:`lib2to3` - 2to3's library
-------------------------------
.. module:: lib2to3
:synopsis: the 2to3 library
.. moduleauthor:: Guido van Rossum
.. moduleauthor:: Collin Winter
.. moduleauthor:: Benjamin Peterson <benjamin@python.org>
**Source code:** :source:`Lib/lib2to3/`
--------------
.. note::
The :mod:`lib2to3` API should be considered unstable and may change
drastically in the future.
.. XXX What is the public interface anyway?

98
third_party/python/Doc/library/__future__.rst vendored Normal file
View file

@ -0,0 +1,98 @@
:mod:`__future__` --- Future statement definitions
==================================================
.. module:: __future__
:synopsis: Future statement definitions
**Source code:** :source:`Lib/__future__.py`
--------------
:mod:`__future__` is a real module, and serves three purposes:
* To avoid confusing existing tools that analyze import statements and expect to
find the modules they're importing.
* To ensure that :ref:`future statements <future>` run under releases prior to
2.1 at least yield runtime exceptions (the import of :mod:`__future__` will
fail, because there was no module of that name prior to 2.1).
* To document when incompatible changes were introduced, and when they will be
--- or were --- made mandatory. This is a form of executable documentation, and
can be inspected programmatically via importing :mod:`__future__` and examining
its contents.
Each statement in :file:`__future__.py` is of the form::
FeatureName = _Feature(OptionalRelease, MandatoryRelease,
CompilerFlag)
where, normally, *OptionalRelease* is less than *MandatoryRelease*, and both are
5-tuples of the same form as :data:`sys.version_info`::
(PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int
PY_MINOR_VERSION, # the 1; an int
PY_MICRO_VERSION, # the 0; an int
PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string
PY_RELEASE_SERIAL # the 3; an int
)
*OptionalRelease* records the first release in which the feature was accepted.
In the case of a *MandatoryRelease* that has not yet occurred,
*MandatoryRelease* predicts the release in which the feature will become part of
the language.
Else *MandatoryRelease* records when the feature became part of the language; in
releases at or after that, modules no longer need a future statement to use the
feature in question, but may continue to use such imports.
*MandatoryRelease* may also be ``None``, meaning that a planned feature got
dropped.
Instances of class :class:`_Feature` have two corresponding methods,
:meth:`getOptionalRelease` and :meth:`getMandatoryRelease`.
*CompilerFlag* is the (bitfield) flag that should be passed in the fourth
argument to the built-in function :func:`compile` to enable the feature in
dynamically compiled code. This flag is stored in the :attr:`compiler_flag`
attribute on :class:`_Feature` instances.
No feature description will ever be deleted from :mod:`__future__`. Since its
introduction in Python 2.1 the following features have found their way into the
language using this mechanism:
+------------------+-------------+--------------+---------------------------------------------+
| feature | optional in | mandatory in | effect |
+==================+=============+==============+=============================================+
| nested_scopes | 2.1.0b1 | 2.2 | :pep:`227`: |
| | | | *Statically Nested Scopes* |
+------------------+-------------+--------------+---------------------------------------------+
| generators | 2.2.0a1 | 2.3 | :pep:`255`: |
| | | | *Simple Generators* |
+------------------+-------------+--------------+---------------------------------------------+
| division | 2.2.0a2 | 3.0 | :pep:`238`: |
| | | | *Changing the Division Operator* |
+------------------+-------------+--------------+---------------------------------------------+
| absolute_import | 2.5.0a1 | 3.0 | :pep:`328`: |
| | | | *Imports: Multi-Line and Absolute/Relative* |
+------------------+-------------+--------------+---------------------------------------------+
| with_statement | 2.5.0a1 | 2.6 | :pep:`343`: |
| | | | *The "with" Statement* |
+------------------+-------------+--------------+---------------------------------------------+
| print_function | 2.6.0a2 | 3.0 | :pep:`3105`: |
| | | | *Make print a function* |
+------------------+-------------+--------------+---------------------------------------------+
| unicode_literals | 2.6.0a2 | 3.0 | :pep:`3112`: |
| | | | *Bytes literals in Python 3000* |
+------------------+-------------+--------------+---------------------------------------------+
| generator_stop | 3.5.0b1 | 3.7 | :pep:`479`: |
| | | | *StopIteration handling inside generators* |
+------------------+-------------+--------------+---------------------------------------------+
.. seealso::
:ref:`future`
How the compiler treats future imports.

25
third_party/python/Doc/library/__main__.rst vendored Normal file
View file

@ -0,0 +1,25 @@
:mod:`__main__` --- Top-level script environment
================================================
.. module:: __main__
:synopsis: The environment where the top-level script is run.
--------------
``'__main__'`` is the name of the scope in which top-level code executes.
A module's __name__ is set equal to ``'__main__'`` when read from
standard input, a script, or from an interactive prompt.
A module can discover whether or not it is running in the main scope by
checking its own ``__name__``, which allows a common idiom for conditionally
executing code in a module when it is run as a script or with ``python
-m`` but not when it is imported::
if __name__ == "__main__":
# execute only if run as a script
main()
For a package, the same effect can be achieved by including a
``__main__.py`` module, the contents of which will be executed when the
module is run with ``-m``.

25
third_party/python/Doc/library/_dummy_thread.rst vendored Normal file
View file

@ -0,0 +1,25 @@
:mod:`_dummy_thread` --- Drop-in replacement for the :mod:`_thread` module
==========================================================================
.. module:: _dummy_thread
:synopsis: Drop-in replacement for the _thread module.
**Source code:** :source:`Lib/_dummy_thread.py`
--------------
This module provides a duplicate interface to the :mod:`_thread` module. It is
meant to be imported when the :mod:`_thread` module is not provided on a
platform.
Suggested usage is::
try:
import _thread
except ImportError:
import _dummy_thread as _thread
Be careful to not use this module where deadlock might occur from a thread being
created that blocks waiting for another thread to be created. This often occurs
with blocking I/O.

192
third_party/python/Doc/library/_thread.rst vendored Normal file
View file

@ -0,0 +1,192 @@
:mod:`_thread` --- Low-level threading API
==========================================
.. module:: _thread
:synopsis: Low-level threading API.
.. index::
single: light-weight processes
single: processes, light-weight
single: binary semaphores
single: semaphores, binary
--------------
This module provides low-level primitives for working with multiple threads
(also called :dfn:`light-weight processes` or :dfn:`tasks`) --- multiple threads of
control sharing their global data space. For synchronization, simple locks
(also called :dfn:`mutexes` or :dfn:`binary semaphores`) are provided.
The :mod:`threading` module provides an easier to use and higher-level
threading API built on top of this module.
.. index::
single: pthreads
pair: threads; POSIX
The module is optional. It is supported on Windows, Linux, SGI IRIX, Solaris
2.x, as well as on systems that have a POSIX thread (a.k.a. "pthread")
implementation. For systems lacking the :mod:`_thread` module, the
:mod:`_dummy_thread` module is available. It duplicates this module's interface
and can be used as a drop-in replacement.
It defines the following constants and functions:
.. exception:: error
Raised on thread-specific errors.
.. versionchanged:: 3.3
This is now a synonym of the built-in :exc:`RuntimeError`.
.. data:: LockType
This is the type of lock objects.
.. function:: start_new_thread(function, args[, kwargs])
Start a new thread and return its identifier. The thread executes the function
*function* with the argument list *args* (which must be a tuple). The optional
*kwargs* argument specifies a dictionary of keyword arguments. When the function
returns, the thread silently exits. When the function terminates with an
unhandled exception, a stack trace is printed and then the thread exits (but
other threads continue to run).
.. function:: interrupt_main()
Raise a :exc:`KeyboardInterrupt` exception in the main thread. A subthread can
use this function to interrupt the main thread.
.. function:: exit()
Raise the :exc:`SystemExit` exception. When not caught, this will cause the
thread to exit silently.
..
function:: exit_prog(status)
Exit all threads and report the value of the integer argument
*status* as the exit status of the entire program.
**Caveat:** code in pending :keyword:`finally` clauses, in this thread
or in other threads, is not executed.
.. function:: allocate_lock()
Return a new lock object. Methods of locks are described below. The lock is
initially unlocked.
.. function:: get_ident()
Return the 'thread identifier' of the current thread. This is a nonzero
integer. Its value has no direct meaning; it is intended as a magic cookie to
be used e.g. to index a dictionary of thread-specific data. Thread identifiers
may be recycled when a thread exits and another thread is created.
.. function:: stack_size([size])
Return the thread stack size used when creating new threads. The optional
*size* argument specifies the stack size to be used for subsequently created
threads, and must be 0 (use platform or configured default) or a positive
integer value of at least 32,768 (32 KiB). If *size* is not specified,
0 is used. If changing the thread stack size is
unsupported, a :exc:`RuntimeError` is raised. If the specified stack size is
invalid, a :exc:`ValueError` is raised and the stack size is unmodified. 32 KiB
is currently the minimum supported stack size value to guarantee sufficient
stack space for the interpreter itself. Note that some platforms may have
particular restrictions on values for the stack size, such as requiring a
minimum stack size > 32 KiB or requiring allocation in multiples of the system
memory page size - platform documentation should be referred to for more
information (4 KiB pages are common; using multiples of 4096 for the stack size is
the suggested approach in the absence of more specific information).
Availability: Windows, systems with POSIX threads.
.. data:: TIMEOUT_MAX
The maximum value allowed for the *timeout* parameter of
:meth:`Lock.acquire`. Specifying a timeout greater than this value will
raise an :exc:`OverflowError`.
.. versionadded:: 3.2
Lock objects have the following methods:
.. method:: lock.acquire(waitflag=1, timeout=-1)
Without any optional argument, this method acquires the lock unconditionally, if
necessary waiting until it is released by another thread (only one thread at a
time can acquire a lock --- that's their reason for existence).
If the integer *waitflag* argument is present, the action depends on its
value: if it is zero, the lock is only acquired if it can be acquired
immediately without waiting, while if it is nonzero, the lock is acquired
unconditionally as above.
If the floating-point *timeout* argument is present and positive, it
specifies the maximum wait time in seconds before returning. A negative
*timeout* argument specifies an unbounded wait. You cannot specify
a *timeout* if *waitflag* is zero.
The return value is ``True`` if the lock is acquired successfully,
``False`` if not.
.. versionchanged:: 3.2
The *timeout* parameter is new.
.. versionchanged:: 3.2
Lock acquires can now be interrupted by signals on POSIX.
.. method:: lock.release()
Releases the lock. The lock must have been acquired earlier, but not
necessarily by the same thread.
.. method:: lock.locked()
Return the status of the lock: ``True`` if it has been acquired by some thread,
``False`` if not.
In addition to these methods, lock objects can also be used via the
:keyword:`with` statement, e.g.::
import _thread
a_lock = _thread.allocate_lock()
with a_lock:
print("a_lock is locked while this executes")
**Caveats:**
.. index:: module: signal
* Threads interact strangely with interrupts: the :exc:`KeyboardInterrupt`
exception will be received by an arbitrary thread. (When the :mod:`signal`
module is available, interrupts always go to the main thread.)
* Calling :func:`sys.exit` or raising the :exc:`SystemExit` exception is
equivalent to calling :func:`_thread.exit`.
* It is not possible to interrupt the :meth:`acquire` method on a lock --- the
:exc:`KeyboardInterrupt` exception will happen after the lock has been acquired.
* When the main thread exits, it is system defined whether the other threads
survive. On most systems, they are killed without executing
:keyword:`try` ... :keyword:`finally` clauses or executing object
destructors.
* When the main thread exits, it does not do any of its usual cleanup (except
that :keyword:`try` ... :keyword:`finally` clauses are honored), and the
standard I/O files are not flushed.

342
third_party/python/Doc/library/abc.rst vendored Normal file
View file

@ -0,0 +1,342 @@
:mod:`abc` --- Abstract Base Classes
====================================
.. module:: abc
:synopsis: Abstract base classes according to PEP 3119.
.. moduleauthor:: Guido van Rossum
.. sectionauthor:: Georg Brandl
.. much of the content adapted from docstrings
**Source code:** :source:`Lib/abc.py`
--------------
This module provides the infrastructure for defining :term:`abstract base
classes <abstract base class>` (ABCs) in Python, as outlined in :pep:`3119`;
see the PEP for why this was added to Python. (See also :pep:`3141` and the
:mod:`numbers` module regarding a type hierarchy for numbers based on ABCs.)
The :mod:`collections` module has some concrete classes that derive from
ABCs; these can, of course, be further derived. In addition, the
:mod:`collections.abc` submodule has some ABCs that can be used to test whether
a class or instance provides a particular interface, for example, if it is
hashable or if it is a mapping.
This module provides the metaclass :class:`ABCMeta` for defining ABCs and
a helper class :class:`ABC` to alternatively define ABCs through inheritance:
.. class:: ABC
A helper class that has :class:`ABCMeta` as its metaclass. With this class,
an abstract base class can be created by simply deriving from :class:`ABC`
avoiding sometimes confusing metaclass usage, for example::
from abc import ABC
class MyABC(ABC):
pass
Note that the type of :class:`ABC` is still :class:`ABCMeta`, therefore
inheriting from :class:`ABC` requires the usual precautions regarding
metaclass usage, as multiple inheritance may lead to metaclass conflicts.
One may also define an abstract base class by passing the metaclass
keyword and using :class:`ABCMeta` directly, for example::
from abc import ABCMeta
class MyABC(metaclass=ABCMeta):
pass
.. versionadded:: 3.4
.. class:: ABCMeta
Metaclass for defining Abstract Base Classes (ABCs).
Use this metaclass to create an ABC. An ABC can be subclassed directly, and
then acts as a mix-in class. You can also register unrelated concrete
classes (even built-in classes) and unrelated ABCs as "virtual subclasses" --
these and their descendants will be considered subclasses of the registering
ABC by the built-in :func:`issubclass` function, but the registering ABC
won't show up in their MRO (Method Resolution Order) nor will method
implementations defined by the registering ABC be callable (not even via
:func:`super`). [#]_
Classes created with a metaclass of :class:`ABCMeta` have the following method:
.. method:: register(subclass)
Register *subclass* as a "virtual subclass" of this ABC. For
example::
from abc import ABC
class MyABC(ABC):
pass
MyABC.register(tuple)
assert issubclass(tuple, MyABC)
assert isinstance((), MyABC)
.. versionchanged:: 3.3
Returns the registered subclass, to allow usage as a class decorator.
.. versionchanged:: 3.4
To detect calls to :meth:`register`, you can use the
:func:`get_cache_token` function.
You can also override this method in an abstract base class:
.. method:: __subclasshook__(subclass)
(Must be defined as a class method.)
Check whether *subclass* is considered a subclass of this ABC. This means
that you can customize the behavior of ``issubclass`` further without the
need to call :meth:`register` on every class you want to consider a
subclass of the ABC. (This class method is called from the
:meth:`__subclasscheck__` method of the ABC.)
This method should return ``True``, ``False`` or ``NotImplemented``. If
it returns ``True``, the *subclass* is considered a subclass of this ABC.
If it returns ``False``, the *subclass* is not considered a subclass of
this ABC, even if it would normally be one. If it returns
``NotImplemented``, the subclass check is continued with the usual
mechanism.
.. XXX explain the "usual mechanism"
For a demonstration of these concepts, look at this example ABC definition::
class Foo:
def __getitem__(self, index):
...
def __len__(self):
...
def get_iterator(self):
return iter(self)
class MyIterable(ABC):
@abstractmethod
def __iter__(self):
while False:
yield None
def get_iterator(self):
return self.__iter__()
@classmethod
def __subclasshook__(cls, C):
if cls is MyIterable:
if any("__iter__" in B.__dict__ for B in C.__mro__):
return True
return NotImplemented
MyIterable.register(Foo)
The ABC ``MyIterable`` defines the standard iterable method,
:meth:`~iterator.__iter__`, as an abstract method. The implementation given
here can still be called from subclasses. The :meth:`get_iterator` method
is also part of the ``MyIterable`` abstract base class, but it does not have
to be overridden in non-abstract derived classes.
The :meth:`__subclasshook__` class method defined here says that any class
that has an :meth:`~iterator.__iter__` method in its
:attr:`~object.__dict__` (or in that of one of its base classes, accessed
via the :attr:`~class.__mro__` list) is considered a ``MyIterable`` too.
Finally, the last line makes ``Foo`` a virtual subclass of ``MyIterable``,
even though it does not define an :meth:`~iterator.__iter__` method (it uses
the old-style iterable protocol, defined in terms of :meth:`__len__` and
:meth:`__getitem__`). Note that this will not make ``get_iterator``
available as a method of ``Foo``, so it is provided separately.
The :mod:`abc` module also provides the following decorator:
.. decorator:: abstractmethod
A decorator indicating abstract methods.
Using this decorator requires that the class's metaclass is :class:`ABCMeta`
or is derived from it. A class that has a metaclass derived from
:class:`ABCMeta` cannot be instantiated unless all of its abstract methods
and properties are overridden. The abstract methods can be called using any
of the normal 'super' call mechanisms. :func:`abstractmethod` may be used
to declare abstract methods for properties and descriptors.
Dynamically adding abstract methods to a class, or attempting to modify the
abstraction status of a method or class once it is created, are not
supported. The :func:`abstractmethod` only affects subclasses derived using
regular inheritance; "virtual subclasses" registered with the ABC's
:meth:`register` method are not affected.
When :func:`abstractmethod` is applied in combination with other method
descriptors, it should be applied as the innermost decorator, as shown in
the following usage examples::
class C(ABC):
@abstractmethod
def my_abstract_method(self, ...):
...
@classmethod
@abstractmethod
def my_abstract_classmethod(cls, ...):
...
@staticmethod
@abstractmethod
def my_abstract_staticmethod(...):
...
@property
@abstractmethod
def my_abstract_property(self):
...
@my_abstract_property.setter
@abstractmethod
def my_abstract_property(self, val):
...
@abstractmethod
def _get_x(self):
...
@abstractmethod
def _set_x(self, val):
...
x = property(_get_x, _set_x)
In order to correctly interoperate with the abstract base class machinery,
the descriptor must identify itself as abstract using
:attr:`__isabstractmethod__`. In general, this attribute should be ``True``
if any of the methods used to compose the descriptor are abstract. For
example, Python's built-in :class:`property` does the equivalent of::
class Descriptor:
...
@property
def __isabstractmethod__(self):
return any(getattr(f, '__isabstractmethod__', False) for
f in (self._fget, self._fset, self._fdel))
.. note::
Unlike Java abstract methods, these abstract
methods may have an implementation. This implementation can be
called via the :func:`super` mechanism from the class that
overrides it. This could be useful as an end-point for a
super-call in a framework that uses cooperative
multiple-inheritance.
The :mod:`abc` module also supports the following legacy decorators:
.. decorator:: abstractclassmethod
.. versionadded:: 3.2
.. deprecated:: 3.3
It is now possible to use :class:`classmethod` with
:func:`abstractmethod`, making this decorator redundant.
A subclass of the built-in :func:`classmethod`, indicating an abstract
classmethod. Otherwise it is similar to :func:`abstractmethod`.
This special case is deprecated, as the :func:`classmethod` decorator
is now correctly identified as abstract when applied to an abstract
method::
class C(ABC):
@classmethod
@abstractmethod
def my_abstract_classmethod(cls, ...):
...
.. decorator:: abstractstaticmethod
.. versionadded:: 3.2
.. deprecated:: 3.3
It is now possible to use :class:`staticmethod` with
:func:`abstractmethod`, making this decorator redundant.
A subclass of the built-in :func:`staticmethod`, indicating an abstract
staticmethod. Otherwise it is similar to :func:`abstractmethod`.
This special case is deprecated, as the :func:`staticmethod` decorator
is now correctly identified as abstract when applied to an abstract
method::
class C(ABC):
@staticmethod
@abstractmethod
def my_abstract_staticmethod(...):
...
.. decorator:: abstractproperty
.. deprecated:: 3.3
It is now possible to use :class:`property`, :meth:`property.getter`,
:meth:`property.setter` and :meth:`property.deleter` with
:func:`abstractmethod`, making this decorator redundant.
A subclass of the built-in :func:`property`, indicating an abstract
property.
This special case is deprecated, as the :func:`property` decorator
is now correctly identified as abstract when applied to an abstract
method::
class C(ABC):
@property
@abstractmethod
def my_abstract_property(self):
...
The above example defines a read-only property; you can also define a
read-write abstract property by appropriately marking one or more of the
underlying methods as abstract::
class C(ABC):
@property
def x(self):
...
@x.setter
@abstractmethod
def x(self, val):
...
If only some components are abstract, only those components need to be
updated to create a concrete property in a subclass::
class D(C):
@C.x.setter
def x(self, val):
...
The :mod:`abc` module also provides the following functions:
.. function:: get_cache_token()
Returns the current abstract base class cache token.
The token is an opaque object (that supports equality testing) identifying
the current version of the abstract base class cache for virtual subclasses.
The token changes with every call to :meth:`ABCMeta.register` on any ABC.
.. versionadded:: 3.4
.. rubric:: Footnotes
.. [#] C++ programmers should note that Python's virtual base class
concept is not the same as C++'s.

239
third_party/python/Doc/library/aifc.rst vendored Normal file
View file

@ -0,0 +1,239 @@
:mod:`aifc` --- Read and write AIFF and AIFC files
==================================================
.. module:: aifc
:synopsis: Read and write audio files in AIFF or AIFC format.
**Source code:** :source:`Lib/aifc.py`
.. index::
single: Audio Interchange File Format
single: AIFF
single: AIFF-C
--------------
This module provides support for reading and writing AIFF and AIFF-C files.
AIFF is Audio Interchange File Format, a format for storing digital audio
samples in a file. AIFF-C is a newer version of the format that includes the
ability to compress the audio data.
Audio files have a number of parameters that describe the audio data. The
sampling rate or frame rate is the number of times per second the sound is
sampled. The number of channels indicate if the audio is mono, stereo, or
quadro. Each frame consists of one sample per channel. The sample size is the
size in bytes of each sample. Thus a frame consists of
``nchannels * samplesize`` bytes, and a second's worth of audio consists of
``nchannels * samplesize * framerate`` bytes.
For example, CD quality audio has a sample size of two bytes (16 bits), uses two
channels (stereo) and has a frame rate of 44,100 frames/second. This gives a
frame size of 4 bytes (2\*2), and a second's worth occupies 2\*2\*44100 bytes
(176,400 bytes).
Module :mod:`aifc` defines the following function:
.. function:: open(file, mode=None)
Open an AIFF or AIFF-C file and return an object instance with methods that are
described below. The argument *file* is either a string naming a file or a
:term:`file object`. *mode* must be ``'r'`` or ``'rb'`` when the file must be
opened for reading, or ``'w'`` or ``'wb'`` when the file must be opened for writing.
If omitted, ``file.mode`` is used if it exists, otherwise ``'rb'`` is used. When
used for writing, the file object should be seekable, unless you know ahead of
time how many samples you are going to write in total and use
:meth:`writeframesraw` and :meth:`setnframes`.
The :func:`.open` function may be used in a :keyword:`with` statement. When
the :keyword:`with` block completes, the :meth:`~aifc.close` method is called.
.. versionchanged:: 3.4
Support for the :keyword:`with` statement was added.
Objects returned by :func:`.open` when a file is opened for reading have the
following methods:
.. method:: aifc.getnchannels()
Return the number of audio channels (1 for mono, 2 for stereo).
.. method:: aifc.getsampwidth()
Return the size in bytes of individual samples.
.. method:: aifc.getframerate()
Return the sampling rate (number of audio frames per second).
.. method:: aifc.getnframes()
Return the number of audio frames in the file.
.. method:: aifc.getcomptype()
Return a bytes array of length 4 describing the type of compression
used in the audio file. For AIFF files, the returned value is
``b'NONE'``.
.. method:: aifc.getcompname()
Return a bytes array convertible to a human-readable description
of the type of compression used in the audio file. For AIFF files,
the returned value is ``b'not compressed'``.
.. method:: aifc.getparams()
Returns a :func:`~collections.namedtuple` ``(nchannels, sampwidth,
framerate, nframes, comptype, compname)``, equivalent to output of the
:meth:`get\*` methods.
.. method:: aifc.getmarkers()
Return a list of markers in the audio file. A marker consists of a tuple of
three elements. The first is the mark ID (an integer), the second is the mark
position in frames from the beginning of the data (an integer), the third is the
name of the mark (a string).
.. method:: aifc.getmark(id)
Return the tuple as described in :meth:`getmarkers` for the mark with the given
*id*.
.. method:: aifc.readframes(nframes)
Read and return the next *nframes* frames from the audio file. The returned
data is a string containing for each frame the uncompressed samples of all
channels.
.. method:: aifc.rewind()
Rewind the read pointer. The next :meth:`readframes` will start from the
beginning.
.. method:: aifc.setpos(pos)
Seek to the specified frame number.
.. method:: aifc.tell()
Return the current frame number.
.. method:: aifc.close()
Close the AIFF file. After calling this method, the object can no longer be
used.
Objects returned by :func:`.open` when a file is opened for writing have all the
above methods, except for :meth:`readframes` and :meth:`setpos`. In addition
the following methods exist. The :meth:`get\*` methods can only be called after
the corresponding :meth:`set\*` methods have been called. Before the first
:meth:`writeframes` or :meth:`writeframesraw`, all parameters except for the
number of frames must be filled in.
.. method:: aifc.aiff()
Create an AIFF file. The default is that an AIFF-C file is created, unless the
name of the file ends in ``'.aiff'`` in which case the default is an AIFF file.
.. method:: aifc.aifc()
Create an AIFF-C file. The default is that an AIFF-C file is created, unless
the name of the file ends in ``'.aiff'`` in which case the default is an AIFF
file.
.. method:: aifc.setnchannels(nchannels)
Specify the number of channels in the audio file.
.. method:: aifc.setsampwidth(width)
Specify the size in bytes of audio samples.
.. method:: aifc.setframerate(rate)
Specify the sampling frequency in frames per second.
.. method:: aifc.setnframes(nframes)
Specify the number of frames that are to be written to the audio file. If this
parameter is not set, or not set correctly, the file needs to support seeking.
.. method:: aifc.setcomptype(type, name)
.. index::
single: u-LAW
single: A-LAW
single: G.722
Specify the compression type. If not specified, the audio data will
not be compressed. In AIFF files, compression is not possible.
The name parameter should be a human-readable description of the
compression type as a bytes array, the type parameter should be a
bytes array of length 4. Currently the following compression types
are supported: ``b'NONE'``, ``b'ULAW'``, ``b'ALAW'``, ``b'G722'``.
.. method:: aifc.setparams(nchannels, sampwidth, framerate, comptype, compname)
Set all the above parameters at once. The argument is a tuple consisting of the
various parameters. This means that it is possible to use the result of a
:meth:`getparams` call as argument to :meth:`setparams`.
.. method:: aifc.setmark(id, pos, name)
Add a mark with the given id (larger than 0), and the given name at the given
position. This method can be called at any time before :meth:`close`.
.. method:: aifc.tell()
Return the current write position in the output file. Useful in combination
with :meth:`setmark`.
.. method:: aifc.writeframes(data)
Write data to the output file. This method can only be called after the audio
file parameters have been set.
.. versionchanged:: 3.4
Any :term:`bytes-like object` is now accepted.
.. method:: aifc.writeframesraw(data)
Like :meth:`writeframes`, except that the header of the audio file is not
updated.
.. versionchanged:: 3.4
Any :term:`bytes-like object` is now accepted.
.. method:: aifc.close()
Close the AIFF file. The header of the file is updated to reflect the actual
size of the audio data. After calling this method, the object can no longer be
used.

29
third_party/python/Doc/library/allos.rst vendored Normal file
View file

@ -0,0 +1,29 @@
.. _allos:
*********************************
Generic Operating System Services
*********************************
The modules described in this chapter provide interfaces to operating system
features that are available on (almost) all operating systems, such as files and
a clock. The interfaces are generally modeled after the Unix or C interfaces,
but they are available on most other systems as well. Here's an overview:
.. toctree::
os.rst
io.rst
time.rst
argparse.rst
getopt.rst
logging.rst
logging.config.rst
logging.handlers.rst
getpass.rst
curses.rst
curses.ascii.rst
curses.panel.rst
platform.rst
errno.rst
ctypes.rst

20
third_party/python/Doc/library/archiving.rst vendored Normal file
View file

@ -0,0 +1,20 @@
.. _archiving:
******************************
Data Compression and Archiving
******************************
The modules described in this chapter support data compression with the zlib,
gzip, bzip2 and lzma algorithms, and the creation of ZIP- and tar-format
archives. See also :ref:`archiving-operations` provided by the :mod:`shutil`
module.
.. toctree::
zlib.rst
gzip.rst
bz2.rst
lzma.rst
zipfile.rst
tarfile.rst

2048
third_party/python/Doc/library/argparse.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

278
third_party/python/Doc/library/array.rst vendored Normal file
View file

@ -0,0 +1,278 @@
:mod:`array` --- Efficient arrays of numeric values
===================================================
.. module:: array
:synopsis: Space efficient arrays of uniformly typed numeric values.
.. index:: single: arrays
--------------
This module defines an object type which can compactly represent an array of
basic values: characters, integers, floating point numbers. Arrays are sequence
types and behave very much like lists, except that the type of objects stored in
them is constrained. The type is specified at object creation time by using a
:dfn:`type code`, which is a single character. The following type codes are
defined:
+-----------+--------------------+-------------------+-----------------------+-------+
| Type code | C Type | Python Type | Minimum size in bytes | Notes |
+===========+====================+===================+=======================+=======+
| ``'b'`` | signed char | int | 1 | |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'B'`` | unsigned char | int | 1 | |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'u'`` | Py_UNICODE | Unicode character | 2 | \(1) |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'h'`` | signed short | int | 2 | |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'H'`` | unsigned short | int | 2 | |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'i'`` | signed int | int | 2 | |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'I'`` | unsigned int | int | 2 | |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'l'`` | signed long | int | 4 | |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'L'`` | unsigned long | int | 4 | |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'q'`` | signed long long | int | 8 | \(2) |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'Q'`` | unsigned long long | int | 8 | \(2) |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'f'`` | float | float | 4 | |
+-----------+--------------------+-------------------+-----------------------+-------+
| ``'d'`` | double | float | 8 | |
+-----------+--------------------+-------------------+-----------------------+-------+
Notes:
(1)
The ``'u'`` type code corresponds to Python's obsolete unicode character
(:c:type:`Py_UNICODE` which is :c:type:`wchar_t`). Depending on the
platform, it can be 16 bits or 32 bits.
``'u'`` will be removed together with the rest of the :c:type:`Py_UNICODE`
API.
.. deprecated-removed:: 3.3 4.0
(2)
The ``'q'`` and ``'Q'`` type codes are available only if
the platform C compiler used to build Python supports C :c:type:`long long`,
or, on Windows, :c:type:`__int64`.
.. versionadded:: 3.3
The actual representation of values is determined by the machine architecture
(strictly speaking, by the C implementation). The actual size can be accessed
through the :attr:`itemsize` attribute.
The module defines the following type:
.. class:: array(typecode[, initializer])
A new array whose items are restricted by *typecode*, and initialized
from the optional *initializer* value, which must be a list, a
:term:`bytes-like object`, or iterable over elements of the
appropriate type.
If given a list or string, the initializer is passed to the new array's
:meth:`fromlist`, :meth:`frombytes`, or :meth:`fromunicode` method (see below)
to add initial items to the array. Otherwise, the iterable initializer is
passed to the :meth:`extend` method.
.. data:: typecodes
A string with all available type codes.
Array objects support the ordinary sequence operations of indexing, slicing,
concatenation, and multiplication. When using slice assignment, the assigned
value must be an array object with the same type code; in all other cases,
:exc:`TypeError` is raised. Array objects also implement the buffer interface,
and may be used wherever :term:`bytes-like objects <bytes-like object>` are supported.
The following data items and methods are also supported:
.. attribute:: array.typecode
The typecode character used to create the array.
.. attribute:: array.itemsize
The length in bytes of one array item in the internal representation.
.. method:: array.append(x)
Append a new item with value *x* to the end of the array.
.. method:: array.buffer_info()
Return a tuple ``(address, length)`` giving the current memory address and the
length in elements of the buffer used to hold array's contents. The size of the
memory buffer in bytes can be computed as ``array.buffer_info()[1] *
array.itemsize``. This is occasionally useful when working with low-level (and
inherently unsafe) I/O interfaces that require memory addresses, such as certain
:c:func:`ioctl` operations. The returned numbers are valid as long as the array
exists and no length-changing operations are applied to it.
.. note::
When using array objects from code written in C or C++ (the only way to
effectively make use of this information), it makes more sense to use the buffer
interface supported by array objects. This method is maintained for backward
compatibility and should be avoided in new code. The buffer interface is
documented in :ref:`bufferobjects`.
.. method:: array.byteswap()
"Byteswap" all items of the array. This is only supported for values which are
1, 2, 4, or 8 bytes in size; for other types of values, :exc:`RuntimeError` is
raised. It is useful when reading data from a file written on a machine with a
different byte order.
.. method:: array.count(x)
Return the number of occurrences of *x* in the array.
.. method:: array.extend(iterable)
Append items from *iterable* to the end of the array. If *iterable* is another
array, it must have *exactly* the same type code; if not, :exc:`TypeError` will
be raised. If *iterable* is not an array, it must be iterable and its elements
must be the right type to be appended to the array.
.. method:: array.frombytes(s)
Appends items from the string, interpreting the string as an array of machine
values (as if it had been read from a file using the :meth:`fromfile` method).
.. versionadded:: 3.2
:meth:`fromstring` is renamed to :meth:`frombytes` for clarity.
.. method:: array.fromfile(f, n)
Read *n* items (as machine values) from the :term:`file object` *f* and append
them to the end of the array. If less than *n* items are available,
:exc:`EOFError` is raised, but the items that were available are still
inserted into the array. *f* must be a real built-in file object; something
else with a :meth:`read` method won't do.
.. method:: array.fromlist(list)
Append items from the list. This is equivalent to ``for x in list:
a.append(x)`` except that if there is a type error, the array is unchanged.
.. method:: array.fromstring()
Deprecated alias for :meth:`frombytes`.
.. method:: array.fromunicode(s)
Extends this array with data from the given unicode string. The array must
be a type ``'u'`` array; otherwise a :exc:`ValueError` is raised. Use
``array.frombytes(unicodestring.encode(enc))`` to append Unicode data to an
array of some other type.
.. method:: array.index(x)
Return the smallest *i* such that *i* is the index of the first occurrence of
*x* in the array.
.. method:: array.insert(i, x)
Insert a new item with value *x* in the array before position *i*. Negative
values are treated as being relative to the end of the array.
.. method:: array.pop([i])
Removes the item with the index *i* from the array and returns it. The optional
argument defaults to ``-1``, so that by default the last item is removed and
returned.
.. method:: array.remove(x)
Remove the first occurrence of *x* from the array.
.. method:: array.reverse()
Reverse the order of the items in the array.
.. method:: array.tobytes()
Convert the array to an array of machine values and return the bytes
representation (the same sequence of bytes that would be written to a file by
the :meth:`tofile` method.)
.. versionadded:: 3.2
:meth:`tostring` is renamed to :meth:`tobytes` for clarity.
.. method:: array.tofile(f)
Write all items (as machine values) to the :term:`file object` *f*.
.. method:: array.tolist()
Convert the array to an ordinary list with the same items.
.. method:: array.tostring()
Deprecated alias for :meth:`tobytes`.
.. method:: array.tounicode()
Convert the array to a unicode string. The array must be a type ``'u'`` array;
otherwise a :exc:`ValueError` is raised. Use ``array.tobytes().decode(enc)`` to
obtain a unicode string from an array of some other type.
When an array object is printed or converted to a string, it is represented as
``array(typecode, initializer)``. The *initializer* is omitted if the array is
empty, otherwise it is a string if the *typecode* is ``'u'``, otherwise it is a
list of numbers. The string is guaranteed to be able to be converted back to an
array with the same type and value using :func:`eval`, so long as the
:class:`~array.array` class has been imported using ``from array import array``.
Examples::
array('l')
array('u', 'hello \u2641')
array('l', [1, 2, 3, 4, 5])
array('d', [1.0, 2.0, 3.14])
.. seealso::
Module :mod:`struct`
Packing and unpacking of heterogeneous binary data.
Module :mod:`xdrlib`
Packing and unpacking of External Data Representation (XDR) data as used in some
remote procedure call systems.
`The Numerical Python Documentation <https://docs.scipy.org/doc/>`_
The Numeric Python extension (NumPy) defines another array type; see
http://www.numpy.org/ for further information about Numerical Python.

270
third_party/python/Doc/library/ast.rst vendored Normal file
View file

@ -0,0 +1,270 @@
:mod:`ast` --- Abstract Syntax Trees
====================================
.. module:: ast
:synopsis: Abstract Syntax Tree classes and manipulation.
.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de>
.. sectionauthor:: Georg Brandl <georg@python.org>
**Source code:** :source:`Lib/ast.py`
--------------
The :mod:`ast` module helps Python applications to process trees of the Python
abstract syntax grammar. The abstract syntax itself might change with each
Python release; this module helps to find out programmatically what the current
grammar looks like.
An abstract syntax tree can be generated by passing :data:`ast.PyCF_ONLY_AST` as
a flag to the :func:`compile` built-in function, or using the :func:`parse`
helper provided in this module. The result will be a tree of objects whose
classes all inherit from :class:`ast.AST`. An abstract syntax tree can be
compiled into a Python code object using the built-in :func:`compile` function.
Node classes
------------
.. class:: AST
This is the base of all AST node classes. The actual node classes are
derived from the :file:`Parser/Python.asdl` file, which is reproduced
:ref:`below <abstract-grammar>`. They are defined in the :mod:`_ast` C
module and re-exported in :mod:`ast`.
There is one class defined for each left-hand side symbol in the abstract
grammar (for example, :class:`ast.stmt` or :class:`ast.expr`). In addition,
there is one class defined for each constructor on the right-hand side; these
classes inherit from the classes for the left-hand side trees. For example,
:class:`ast.BinOp` inherits from :class:`ast.expr`. For production rules
with alternatives (aka "sums"), the left-hand side class is abstract: only
instances of specific constructor nodes are ever created.
.. index:: single: ? (question mark); in AST grammar
.. index:: single: * (asterisk); in AST grammar
.. attribute:: _fields
Each concrete class has an attribute :attr:`_fields` which gives the names
of all child nodes.
Each instance of a concrete class has one attribute for each child node,
of the type as defined in the grammar. For example, :class:`ast.BinOp`
instances have an attribute :attr:`left` of type :class:`ast.expr`.
If these attributes are marked as optional in the grammar (using a
question mark), the value might be ``None``. If the attributes can have
zero-or-more values (marked with an asterisk), the values are represented
as Python lists. All possible attributes must be present and have valid
values when compiling an AST with :func:`compile`.
.. attribute:: lineno
col_offset
Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have
:attr:`lineno` and :attr:`col_offset` attributes. The :attr:`lineno` is
the line number of source text (1-indexed so the first line is line 1) and
the :attr:`col_offset` is the UTF-8 byte offset of the first token that
generated the node. The UTF-8 offset is recorded because the parser uses
UTF-8 internally.
The constructor of a class :class:`ast.T` parses its arguments as follows:
* If there are positional arguments, there must be as many as there are items
in :attr:`T._fields`; they will be assigned as attributes of these names.
* If there are keyword arguments, they will set the attributes of the same
names to the given values.
For example, to create and populate an :class:`ast.UnaryOp` node, you could
use ::
node = ast.UnaryOp()
node.op = ast.USub()
node.operand = ast.Num()
node.operand.n = 5
node.operand.lineno = 0
node.operand.col_offset = 0
node.lineno = 0
node.col_offset = 0
or the more compact ::
node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0),
lineno=0, col_offset=0)
.. _abstract-grammar:
Abstract Grammar
----------------
The abstract grammar is currently defined as follows:
.. literalinclude:: ../../Parser/Python.asdl
:language: none
:mod:`ast` Helpers
------------------
Apart from the node classes, the :mod:`ast` module defines these utility functions
and classes for traversing abstract syntax trees:
.. function:: parse(source, filename='<unknown>', mode='exec')
Parse the source into an AST node. Equivalent to ``compile(source,
filename, mode, ast.PyCF_ONLY_AST)``.
.. warning::
It is possible to crash the Python interpreter with a
sufficiently large/complex string due to stack depth limitations
in Python's AST compiler.
.. function:: literal_eval(node_or_string)
Safely evaluate an expression node or a string containing a Python literal or
container display. The string or node provided may only consist of the
following Python literal structures: strings, bytes, numbers, tuples, lists,
dicts, sets, booleans, and ``None``.
This can be used for safely evaluating strings containing Python values from
untrusted sources without the need to parse the values oneself. It is not
capable of evaluating arbitrarily complex expressions, for example involving
operators or indexing.
.. warning::
It is possible to crash the Python interpreter with a
sufficiently large/complex string due to stack depth limitations
in Python's AST compiler.
.. versionchanged:: 3.2
Now allows bytes and set literals.
.. function:: get_docstring(node, clean=True)
Return the docstring of the given *node* (which must be a
:class:`FunctionDef`, :class:`ClassDef` or :class:`Module` node), or ``None``
if it has no docstring. If *clean* is true, clean up the docstring's
indentation with :func:`inspect.cleandoc`.
.. function:: fix_missing_locations(node)
When you compile a node tree with :func:`compile`, the compiler expects
:attr:`lineno` and :attr:`col_offset` attributes for every node that supports
them. This is rather tedious to fill in for generated nodes, so this helper
adds these attributes recursively where not already set, by setting them to
the values of the parent node. It works recursively starting at *node*.
.. function:: increment_lineno(node, n=1)
Increment the line number of each node in the tree starting at *node* by *n*.
This is useful to "move code" to a different location in a file.
.. function:: copy_location(new_node, old_node)
Copy source location (:attr:`lineno` and :attr:`col_offset`) from *old_node*
to *new_node* if possible, and return *new_node*.
.. function:: iter_fields(node)
Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields``
that is present on *node*.
.. function:: iter_child_nodes(node)
Yield all direct child nodes of *node*, that is, all fields that are nodes
and all items of fields that are lists of nodes.
.. function:: walk(node)
Recursively yield all descendant nodes in the tree starting at *node*
(including *node* itself), in no specified order. This is useful if you only
want to modify nodes in place and don't care about the context.
.. class:: NodeVisitor()
A node visitor base class that walks the abstract syntax tree and calls a
visitor function for every node found. This function may return a value
which is forwarded by the :meth:`visit` method.
This class is meant to be subclassed, with the subclass adding visitor
methods.
.. method:: visit(node)
Visit a node. The default implementation calls the method called
:samp:`self.visit_{classname}` where *classname* is the name of the node
class, or :meth:`generic_visit` if that method doesn't exist.
.. method:: generic_visit(node)
This visitor calls :meth:`visit` on all children of the node.
Note that child nodes of nodes that have a custom visitor method won't be
visited unless the visitor calls :meth:`generic_visit` or visits them
itself.
Don't use the :class:`NodeVisitor` if you want to apply changes to nodes
during traversal. For this a special visitor exists
(:class:`NodeTransformer`) that allows modifications.
.. class:: NodeTransformer()
A :class:`NodeVisitor` subclass that walks the abstract syntax tree and
allows modification of nodes.
The :class:`NodeTransformer` will walk the AST and use the return value of
the visitor methods to replace or remove the old node. If the return value
of the visitor method is ``None``, the node will be removed from its
location, otherwise it is replaced with the return value. The return value
may be the original node in which case no replacement takes place.
Here is an example transformer that rewrites all occurrences of name lookups
(``foo``) to ``data['foo']``::
class RewriteName(NodeTransformer):
def visit_Name(self, node):
return copy_location(Subscript(
value=Name(id='data', ctx=Load()),
slice=Index(value=Str(s=node.id)),
ctx=node.ctx
), node)
Keep in mind that if the node you're operating on has child nodes you must
either transform the child nodes yourself or call the :meth:`generic_visit`
method for the node first.
For nodes that were part of a collection of statements (that applies to all
statement nodes), the visitor may also return a list of nodes rather than
just a single node.
Usually you use the transformer like this::
node = YourTransformer().visit(node)
.. function:: dump(node, annotate_fields=True, include_attributes=False)
Return a formatted dump of the tree in *node*. This is mainly useful for
debugging purposes. The returned string will show the names and the values
for fields. This makes the code impossible to evaluate, so if evaluation is
wanted *annotate_fields* must be set to ``False``. Attributes such as line
numbers and column offsets are not dumped by default. If this is wanted,
*include_attributes* can be set to ``True``.
.. seealso::
`Green Tree Snakes <https://greentreesnakes.readthedocs.org/>`_, an external documentation resource, has good
details on working with Python ASTs.

213
third_party/python/Doc/library/asynchat.rst vendored Normal file
View file

@ -0,0 +1,213 @@
:mod:`asynchat` --- Asynchronous socket command/response handler
================================================================
.. module:: asynchat
:synopsis: Support for asynchronous command/response protocols.
.. moduleauthor:: Sam Rushing <rushing@nightmare.com>
.. sectionauthor:: Steve Holden <sholden@holdenweb.com>
**Source code:** :source:`Lib/asynchat.py`
.. deprecated:: 3.6
Please use :mod:`asyncio` instead.
--------------
.. note::
This module exists for backwards compatibility only. For new code we
recommend using :mod:`asyncio`.
This module builds on the :mod:`asyncore` infrastructure, simplifying
asynchronous clients and servers and making it easier to handle protocols
whose elements are terminated by arbitrary strings, or are of variable length.
:mod:`asynchat` defines the abstract class :class:`async_chat` that you
subclass, providing implementations of the :meth:`collect_incoming_data` and
:meth:`found_terminator` methods. It uses the same asynchronous loop as
:mod:`asyncore`, and the two types of channel, :class:`asyncore.dispatcher`
and :class:`asynchat.async_chat`, can freely be mixed in the channel map.
Typically an :class:`asyncore.dispatcher` server channel generates new
:class:`asynchat.async_chat` channel objects as it receives incoming
connection requests.
.. class:: async_chat()
This class is an abstract subclass of :class:`asyncore.dispatcher`. To make
practical use of the code you must subclass :class:`async_chat`, providing
meaningful :meth:`collect_incoming_data` and :meth:`found_terminator`
methods.
The :class:`asyncore.dispatcher` methods can be used, although not all make
sense in a message/response context.
Like :class:`asyncore.dispatcher`, :class:`async_chat` defines a set of
events that are generated by an analysis of socket conditions after a
:c:func:`select` call. Once the polling loop has been started the
:class:`async_chat` object's methods are called by the event-processing
framework with no action on the part of the programmer.
Two class attributes can be modified, to improve performance, or possibly
even to conserve memory.
.. data:: ac_in_buffer_size
The asynchronous input buffer size (default ``4096``).
.. data:: ac_out_buffer_size
The asynchronous output buffer size (default ``4096``).
Unlike :class:`asyncore.dispatcher`, :class:`async_chat` allows you to
define a :abbr:`FIFO (first-in, first-out)` queue of *producers*. A producer need
have only one method, :meth:`more`, which should return data to be
transmitted on the channel.
The producer indicates exhaustion (*i.e.* that it contains no more data) by
having its :meth:`more` method return the empty bytes object. At this point
the :class:`async_chat` object removes the producer from the queue and starts
using the next producer, if any. When the producer queue is empty the
:meth:`handle_write` method does nothing. You use the channel object's
:meth:`set_terminator` method to describe how to recognize the end of, or
an important breakpoint in, an incoming transmission from the remote
endpoint.
To build a functioning :class:`async_chat` subclass your input methods
:meth:`collect_incoming_data` and :meth:`found_terminator` must handle the
data that the channel receives asynchronously. The methods are described
below.
.. method:: async_chat.close_when_done()
Pushes a ``None`` on to the producer queue. When this producer is popped off
the queue it causes the channel to be closed.
.. method:: async_chat.collect_incoming_data(data)
Called with *data* holding an arbitrary amount of received data. The
default method, which must be overridden, raises a
:exc:`NotImplementedError` exception.
.. method:: async_chat.discard_buffers()
In emergencies this method will discard any data held in the input and/or
output buffers and the producer queue.
.. method:: async_chat.found_terminator()
Called when the incoming data stream matches the termination condition set
by :meth:`set_terminator`. The default method, which must be overridden,
raises a :exc:`NotImplementedError` exception. The buffered input data
should be available via an instance attribute.
.. method:: async_chat.get_terminator()
Returns the current terminator for the channel.
.. method:: async_chat.push(data)
Pushes data on to the channel's queue to ensure its transmission.
This is all you need to do to have the channel write the data out to the
network, although it is possible to use your own producers in more complex
schemes to implement encryption and chunking, for example.
.. method:: async_chat.push_with_producer(producer)
Takes a producer object and adds it to the producer queue associated with
the channel. When all currently-pushed producers have been exhausted the
channel will consume this producer's data by calling its :meth:`more`
method and send the data to the remote endpoint.
.. method:: async_chat.set_terminator(term)
Sets the terminating condition to be recognized on the channel. ``term``
may be any of three types of value, corresponding to three different ways
to handle incoming protocol data.
+-----------+---------------------------------------------+
| term | Description |
+===========+=============================================+
| *string* | Will call :meth:`found_terminator` when the |
| | string is found in the input stream |
+-----------+---------------------------------------------+
| *integer* | Will call :meth:`found_terminator` when the |
| | indicated number of characters have been |
| | received |
+-----------+---------------------------------------------+
| ``None`` | The channel continues to collect data |
| | forever |
+-----------+---------------------------------------------+
Note that any data following the terminator will be available for reading
by the channel after :meth:`found_terminator` is called.
.. _asynchat-example:
asynchat Example
----------------
The following partial example shows how HTTP requests can be read with
:class:`async_chat`. A web server might create an
:class:`http_request_handler` object for each incoming client connection.
Notice that initially the channel terminator is set to match the blank line at
the end of the HTTP headers, and a flag indicates that the headers are being
read.
Once the headers have been read, if the request is of type POST (indicating
that further data are present in the input stream) then the
``Content-Length:`` header is used to set a numeric terminator to read the
right amount of data from the channel.
The :meth:`handle_request` method is called once all relevant input has been
marshalled, after setting the channel terminator to ``None`` to ensure that
any extraneous data sent by the web client are ignored. ::
import asynchat
class http_request_handler(asynchat.async_chat):
def __init__(self, sock, addr, sessions, log):
asynchat.async_chat.__init__(self, sock=sock)
self.addr = addr
self.sessions = sessions
self.ibuffer = []
self.obuffer = b""
self.set_terminator(b"\r\n\r\n")
self.reading_headers = True
self.handling = False
self.cgi_data = None
self.log = log
def collect_incoming_data(self, data):
"""Buffer the data"""
self.ibuffer.append(data)
def found_terminator(self):
if self.reading_headers:
self.reading_headers = False
self.parse_headers(b"".join(self.ibuffer))
self.ibuffer = []
if self.op.upper() == b"POST":
clen = self.headers.getheader("content-length")
self.set_terminator(int(clen))
else:
self.handling = True
self.set_terminator(None)
self.handle_request()
elif not self.handling:
self.set_terminator(None) # browsers sometimes over-send
self.cgi_data = parse(self.headers, b"".join(self.ibuffer))
self.handling = True
self.ibuffer = []
self.handle_request()

418
third_party/python/Doc/library/asyncio-dev.rst vendored Normal file
View file

@ -0,0 +1,418 @@
.. currentmodule:: asyncio
.. _asyncio-dev:
Develop with asyncio
====================
Asynchronous programming is different than classical "sequential" programming.
This page lists common traps and explains how to avoid them.
.. _asyncio-debug-mode:
Debug mode of asyncio
---------------------
The implementation of :mod:`asyncio` has been written for performance.
In order to ease the development of asynchronous code, you may wish to
enable *debug mode*.
To enable all debug checks for an application:
* Enable the asyncio debug mode globally by setting the environment variable
:envvar:`PYTHONASYNCIODEBUG` to ``1``, or by calling :meth:`AbstractEventLoop.set_debug`.
* Set the log level of the :ref:`asyncio logger <asyncio-logger>` to
:py:data:`logging.DEBUG`. For example, call
``logging.basicConfig(level=logging.DEBUG)`` at startup.
* Configure the :mod:`warnings` module to display :exc:`ResourceWarning`
warnings. For example, use the ``-Wdefault`` command line option of Python to
display them.
Examples debug checks:
* Log :ref:`coroutines defined but never "yielded from"
<asyncio-coroutine-not-scheduled>`
* :meth:`~AbstractEventLoop.call_soon` and :meth:`~AbstractEventLoop.call_at` methods
raise an exception if they are called from the wrong thread.
* Log the execution time of the selector
* Log callbacks taking more than 100 ms to be executed. The
:attr:`AbstractEventLoop.slow_callback_duration` attribute is the minimum
duration in seconds of "slow" callbacks.
* :exc:`ResourceWarning` warnings are emitted when transports and event loops
are :ref:`not closed explicitly <asyncio-close-transports>`.
.. seealso::
The :meth:`AbstractEventLoop.set_debug` method and the :ref:`asyncio logger
<asyncio-logger>`.
Cancellation
------------
Cancellation of tasks is not common in classic programming. In asynchronous
programming, not only is it something common, but you have to prepare your
code to handle it.
Futures and tasks can be cancelled explicitly with their :meth:`Future.cancel`
method. The :func:`wait_for` function cancels the waited task when the timeout
occurs. There are many other cases where a task can be cancelled indirectly.
Don't call :meth:`~Future.set_result` or :meth:`~Future.set_exception` method
of :class:`Future` if the future is cancelled: it would fail with an exception.
For example, write::
if not fut.cancelled():
fut.set_result('done')
Don't schedule directly a call to the :meth:`~Future.set_result` or the
:meth:`~Future.set_exception` method of a future with
:meth:`AbstractEventLoop.call_soon`: the future can be cancelled before its method
is called.
If you wait for a future, you should check early if the future was cancelled to
avoid useless operations. Example::
@coroutine
def slow_operation(fut):
if fut.cancelled():
return
# ... slow computation ...
yield from fut
# ...
The :func:`shield` function can also be used to ignore cancellation.
.. _asyncio-multithreading:
Concurrency and multithreading
------------------------------
An event loop runs in a thread and executes all callbacks and tasks in the same
thread. While a task is running in the event loop, no other task is running in
the same thread. But when the task uses ``yield from``, the task is suspended
and the event loop executes the next task.
To schedule a callback from a different thread, the
:meth:`AbstractEventLoop.call_soon_threadsafe` method should be used. Example::
loop.call_soon_threadsafe(callback, *args)
Most asyncio objects are not thread safe. You should only worry if you access
objects outside the event loop. For example, to cancel a future, don't call
directly its :meth:`Future.cancel` method, but::
loop.call_soon_threadsafe(fut.cancel)
To handle signals and to execute subprocesses, the event loop must be run in
the main thread.
To schedule a coroutine object from a different thread, the
:func:`run_coroutine_threadsafe` function should be used. It returns a
:class:`concurrent.futures.Future` to access the result::
future = asyncio.run_coroutine_threadsafe(coro_func(), loop)
result = future.result(timeout) # Wait for the result with a timeout
The :meth:`AbstractEventLoop.run_in_executor` method can be used with a thread pool
executor to execute a callback in different thread to not block the thread of
the event loop.
.. seealso::
The :ref:`Synchronization primitives <asyncio-sync>` section describes ways
to synchronize tasks.
The :ref:`Subprocess and threads <asyncio-subprocess-threads>` section lists
asyncio limitations to run subprocesses from different threads.
.. _asyncio-handle-blocking:
Handle blocking functions correctly
-----------------------------------
Blocking functions should not be called directly. For example, if a function
blocks for 1 second, other tasks are delayed by 1 second which can have an
important impact on reactivity.
For networking and subprocesses, the :mod:`asyncio` module provides high-level
APIs like :ref:`protocols <asyncio-protocol>`.
An executor can be used to run a task in a different thread or even in a
different process, to not block the thread of the event loop. See the
:meth:`AbstractEventLoop.run_in_executor` method.
.. seealso::
The :ref:`Delayed calls <asyncio-delayed-calls>` section details how the
event loop handles time.
.. _asyncio-logger:
Logging
-------
The :mod:`asyncio` module logs information with the :mod:`logging` module in
the logger ``'asyncio'``.
The default log level for the :mod:`asyncio` module is :py:data:`logging.INFO`.
For those not wanting such verbosity from :mod:`asyncio` the log level can
be changed. For example, to change the level to :py:data:`logging.WARNING`:
.. code-block:: none
logging.getLogger('asyncio').setLevel(logging.WARNING)
.. _asyncio-coroutine-not-scheduled:
Detect coroutine objects never scheduled
----------------------------------------
When a coroutine function is called and its result is not passed to
:func:`ensure_future` or to the :meth:`AbstractEventLoop.create_task` method,
the execution of the coroutine object will never be scheduled which is
probably a bug. :ref:`Enable the debug mode of asyncio <asyncio-debug-mode>`
to :ref:`log a warning <asyncio-logger>` to detect it.
Example with the bug::
import asyncio
@asyncio.coroutine
def test():
print("never scheduled")
test()
Output in debug mode::
Coroutine test() at test.py:3 was never yielded from
Coroutine object created at (most recent call last):
File "test.py", line 7, in <module>
test()
The fix is to call the :func:`ensure_future` function or the
:meth:`AbstractEventLoop.create_task` method with the coroutine object.
.. seealso::
:ref:`Pending task destroyed <asyncio-pending-task-destroyed>`.
Detect exceptions never consumed
--------------------------------
Python usually calls :func:`sys.excepthook` on unhandled exceptions. If
:meth:`Future.set_exception` is called, but the exception is never consumed,
:func:`sys.excepthook` is not called. Instead, :ref:`a log is emitted
<asyncio-logger>` when the future is deleted by the garbage collector, with the
traceback where the exception was raised.
Example of unhandled exception::
import asyncio
@asyncio.coroutine
def bug():
raise Exception("not consumed")
loop = asyncio.get_event_loop()
asyncio.ensure_future(bug())
loop.run_forever()
loop.close()
Output::
Task exception was never retrieved
future: <Task finished coro=<coro() done, defined at asyncio/coroutines.py:139> exception=Exception('not consumed',)>
Traceback (most recent call last):
File "asyncio/tasks.py", line 237, in _step
result = next(coro)
File "asyncio/coroutines.py", line 141, in coro
res = func(*args, **kw)
File "test.py", line 5, in bug
raise Exception("not consumed")
Exception: not consumed
:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the
traceback where the task was created. Output in debug mode::
Task exception was never retrieved
future: <Task finished coro=<bug() done, defined at test.py:3> exception=Exception('not consumed',) created at test.py:8>
source_traceback: Object created at (most recent call last):
File "test.py", line 8, in <module>
asyncio.ensure_future(bug())
Traceback (most recent call last):
File "asyncio/tasks.py", line 237, in _step
result = next(coro)
File "asyncio/coroutines.py", line 79, in __next__
return next(self.gen)
File "asyncio/coroutines.py", line 141, in coro
res = func(*args, **kw)
File "test.py", line 5, in bug
raise Exception("not consumed")
Exception: not consumed
There are different options to fix this issue. The first option is to chain the
coroutine in another coroutine and use classic try/except::
@asyncio.coroutine
def handle_exception():
try:
yield from bug()
except Exception:
print("exception consumed")
loop = asyncio.get_event_loop()
asyncio.ensure_future(handle_exception())
loop.run_forever()
loop.close()
Another option is to use the :meth:`AbstractEventLoop.run_until_complete`
function::
task = asyncio.ensure_future(bug())
try:
loop.run_until_complete(task)
except Exception:
print("exception consumed")
.. seealso::
The :meth:`Future.exception` method.
Chain coroutines correctly
--------------------------
When a coroutine function calls other coroutine functions and tasks, they
should be chained explicitly with ``yield from``. Otherwise, the execution is
not guaranteed to be sequential.
Example with different bugs using :func:`asyncio.sleep` to simulate slow
operations::
import asyncio
@asyncio.coroutine
def create():
yield from asyncio.sleep(3.0)
print("(1) create file")
@asyncio.coroutine
def write():
yield from asyncio.sleep(1.0)
print("(2) write into file")
@asyncio.coroutine
def close():
print("(3) close file")
@asyncio.coroutine
def test():
asyncio.ensure_future(create())
asyncio.ensure_future(write())
asyncio.ensure_future(close())
yield from asyncio.sleep(2.0)
loop.stop()
loop = asyncio.get_event_loop()
asyncio.ensure_future(test())
loop.run_forever()
print("Pending tasks at exit: %s" % asyncio.Task.all_tasks(loop))
loop.close()
Expected output:
.. code-block:: none
(1) create file
(2) write into file
(3) close file
Pending tasks at exit: set()
Actual output:
.. code-block:: none
(3) close file
(2) write into file
Pending tasks at exit: {<Task pending create() at test.py:7 wait_for=<Future pending cb=[Task._wakeup()]>>}
Task was destroyed but it is pending!
task: <Task pending create() done at test.py:5 wait_for=<Future pending cb=[Task._wakeup()]>>
The loop stopped before the ``create()`` finished, ``close()`` has been called
before ``write()``, whereas coroutine functions were called in this order:
``create()``, ``write()``, ``close()``.
To fix the example, tasks must be marked with ``yield from``::
@asyncio.coroutine
def test():
yield from asyncio.ensure_future(create())
yield from asyncio.ensure_future(write())
yield from asyncio.ensure_future(close())
yield from asyncio.sleep(2.0)
loop.stop()
Or without ``asyncio.ensure_future()``::
@asyncio.coroutine
def test():
yield from create()
yield from write()
yield from close()
yield from asyncio.sleep(2.0)
loop.stop()
.. _asyncio-pending-task-destroyed:
Pending task destroyed
----------------------
If a pending task is destroyed, the execution of its wrapped :ref:`coroutine
<coroutine>` did not complete. It is probably a bug and so a warning is logged.
Example of log:
.. code-block:: none
Task was destroyed but it is pending!
task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()]>>
:ref:`Enable the debug mode of asyncio <asyncio-debug-mode>` to get the
traceback where the task was created. Example of log in debug mode:
.. code-block:: none
Task was destroyed but it is pending!
source_traceback: Object created at (most recent call last):
File "test.py", line 15, in <module>
task = asyncio.ensure_future(coro, loop=loop)
task: <Task pending coro=<kill_me() done, defined at test.py:5> wait_for=<Future pending cb=[Task._wakeup()] created at test.py:7> created at test.py:15>
.. seealso::
:ref:`Detect coroutine objects never scheduled <asyncio-coroutine-not-scheduled>`.
.. _asyncio-close-transports:
Close transports and event loops
--------------------------------
When a transport is no more needed, call its ``close()`` method to release
resources. Event loops must also be closed explicitly.
If a transport or an event loop is not closed explicitly, a
:exc:`ResourceWarning` warning will be emitted in its destructor. By default,
:exc:`ResourceWarning` warnings are ignored. The :ref:`Debug mode of asyncio
<asyncio-debug-mode>` section explains how to display them.

1025
third_party/python/Doc/library/asyncio-eventloop.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

232
third_party/python/Doc/library/asyncio-eventloops.rst vendored Normal file
View file

@ -0,0 +1,232 @@
.. currentmodule:: asyncio
Event loops
===========
**Source code:** :source:`Lib/asyncio/events.py`
Event loop functions
--------------------
The following functions are convenient shortcuts to accessing the methods of the
global policy. Note that this provides access to the default policy, unless an
alternative policy was set by calling :func:`set_event_loop_policy` earlier in
the execution of the process.
.. function:: get_event_loop()
Equivalent to calling ``get_event_loop_policy().get_event_loop()``.
.. function:: set_event_loop(loop)
Equivalent to calling ``get_event_loop_policy().set_event_loop(loop)``.
.. function:: new_event_loop()
Equivalent to calling ``get_event_loop_policy().new_event_loop()``.
.. _asyncio-event-loops:
Available event loops
---------------------
asyncio currently provides two implementations of event loops:
:class:`SelectorEventLoop` and :class:`ProactorEventLoop`.
.. class:: SelectorEventLoop
Event loop based on the :mod:`selectors` module. Subclass of
:class:`AbstractEventLoop`.
Use the most efficient selector available on the platform.
On Windows, only sockets are supported (ex: pipes are not supported):
see the `MSDN documentation of select
<https://msdn.microsoft.com/en-us/library/windows/desktop/ms740141%28v=vs.85%29.aspx>`_.
.. class:: ProactorEventLoop
Proactor event loop for Windows using "I/O Completion Ports" aka IOCP.
Subclass of :class:`AbstractEventLoop`.
Availability: Windows.
.. seealso::
`MSDN documentation on I/O Completion Ports
<https://msdn.microsoft.com/en-us/library/windows/desktop/aa365198%28v=vs.85%29.aspx>`_.
Example to use a :class:`ProactorEventLoop` on Windows::
import asyncio, sys
if sys.platform == 'win32':
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
.. _asyncio-platform-support:
Platform support
----------------
The :mod:`asyncio` module has been designed to be portable, but each platform
still has subtle differences and may not support all :mod:`asyncio` features.
Windows
^^^^^^^
Common limits of Windows event loops:
- :meth:`~AbstractEventLoop.create_unix_connection` and
:meth:`~AbstractEventLoop.create_unix_server` are not supported: the socket
family :data:`socket.AF_UNIX` is specific to UNIX
- :meth:`~AbstractEventLoop.add_signal_handler` and
:meth:`~AbstractEventLoop.remove_signal_handler` are not supported
- :meth:`EventLoopPolicy.set_child_watcher` is not supported.
:class:`ProactorEventLoop` supports subprocesses. It has only one
implementation to watch child processes, there is no need to configure it.
:class:`SelectorEventLoop` specific limits:
- :class:`~selectors.SelectSelector` is used which only supports sockets
and is limited to 512 sockets.
- :meth:`~AbstractEventLoop.add_reader` and :meth:`~AbstractEventLoop.add_writer` only
accept file descriptors of sockets
- Pipes are not supported
(ex: :meth:`~AbstractEventLoop.connect_read_pipe`,
:meth:`~AbstractEventLoop.connect_write_pipe`)
- :ref:`Subprocesses <asyncio-subprocess>` are not supported
(ex: :meth:`~AbstractEventLoop.subprocess_exec`,
:meth:`~AbstractEventLoop.subprocess_shell`)
:class:`ProactorEventLoop` specific limits:
- :meth:`~AbstractEventLoop.create_datagram_endpoint` (UDP) is not supported
- :meth:`~AbstractEventLoop.add_reader` and :meth:`~AbstractEventLoop.add_writer` are
not supported
The resolution of the monotonic clock on Windows is usually around 15.6 msec.
The best resolution is 0.5 msec. The resolution depends on the hardware
(availability of `HPET
<https://en.wikipedia.org/wiki/High_Precision_Event_Timer>`_) and on the Windows
configuration. See :ref:`asyncio delayed calls <asyncio-delayed-calls>`.
.. versionchanged:: 3.5
:class:`ProactorEventLoop` now supports SSL.
Mac OS X
^^^^^^^^
Character devices like PTY are only well supported since Mavericks (Mac OS
10.9). They are not supported at all on Mac OS 10.5 and older.
On Mac OS 10.6, 10.7 and 10.8, the default event loop is
:class:`SelectorEventLoop` which uses :class:`selectors.KqueueSelector`.
:class:`selectors.KqueueSelector` does not support character devices on these
versions. The :class:`SelectorEventLoop` can be used with
:class:`~selectors.SelectSelector` or :class:`~selectors.PollSelector` to
support character devices on these versions of Mac OS X. Example::
import asyncio
import selectors
selector = selectors.SelectSelector()
loop = asyncio.SelectorEventLoop(selector)
asyncio.set_event_loop(loop)
Event loop policies and the default policy
------------------------------------------
Event loop management is abstracted with a *policy* pattern, to provide maximal
flexibility for custom platforms and frameworks. Throughout the execution of a
process, a single global policy object manages the event loops available to the
process based on the calling context. A policy is an object implementing the
:class:`AbstractEventLoopPolicy` interface.
For most users of :mod:`asyncio`, policies never have to be dealt with
explicitly, since the default global policy is sufficient (see below).
The module-level functions
:func:`get_event_loop` and :func:`set_event_loop` provide convenient access to
event loops managed by the default policy.
Event loop policy interface
---------------------------
An event loop policy must implement the following interface:
.. class:: AbstractEventLoopPolicy
Event loop policy.
.. method:: get_event_loop()
Get the event loop for the current context.
Returns an event loop object implementing the :class:`AbstractEventLoop`
interface. In case called from coroutine, it returns the currently
running event loop.
Raises an exception in case no event loop has been set for the current
context and the current policy does not specify to create one. It must
never return ``None``.
.. versionchanged:: 3.6
.. method:: set_event_loop(loop)
Set the event loop for the current context to *loop*.
.. method:: new_event_loop()
Create and return a new event loop object according to this policy's
rules.
If there's need to set this loop as the event loop for the current
context, :meth:`set_event_loop` must be called explicitly.
The default policy defines context as the current thread, and manages an event
loop per thread that interacts with :mod:`asyncio`. If the current thread
doesn't already have an event loop associated with it, the default policy's
:meth:`~AbstractEventLoopPolicy.get_event_loop` method creates one when
called from the main thread, but raises :exc:`RuntimeError` otherwise.
Access to the global loop policy
--------------------------------
.. function:: get_event_loop_policy()
Get the current event loop policy.
.. function:: set_event_loop_policy(policy)
Set the current event loop policy. If *policy* is ``None``, the default
policy is restored.
Customizing the event loop policy
---------------------------------
To implement a new event loop policy, it is recommended you subclass the
concrete default event loop policy :class:`DefaultEventLoopPolicy`
and override the methods for which you want to change behavior, for example::
class MyEventLoopPolicy(asyncio.DefaultEventLoopPolicy):
def get_event_loop(self):
"""Get the event loop.
This may be None or an instance of EventLoop.
"""
loop = super().get_event_loop()
# Do something with loop ...
return loop
asyncio.set_event_loop_policy(MyEventLoopPolicy())

750
third_party/python/Doc/library/asyncio-protocol.rst vendored Normal file
View file

@ -0,0 +1,750 @@
.. currentmodule:: asyncio
+++++++++++++++++++++++++++++++++++++++++++++
Transports and protocols (callback based API)
+++++++++++++++++++++++++++++++++++++++++++++
**Source code:** :source:`Lib/asyncio/transports.py`
**Source code:** :source:`Lib/asyncio/protocols.py`
.. _asyncio-transport:
Transports
==========
Transports are classes provided by :mod:`asyncio` in order to abstract
various kinds of communication channels. You generally won't instantiate
a transport yourself; instead, you will call an :class:`AbstractEventLoop` method
which will create the transport and try to initiate the underlying
communication channel, calling you back when it succeeds.
Once the communication channel is established, a transport is always
paired with a :ref:`protocol <asyncio-protocol>` instance. The protocol can
then call the transport's methods for various purposes.
:mod:`asyncio` currently implements transports for TCP, UDP, SSL, and
subprocess pipes. The methods available on a transport depend on
the transport's kind.
The transport classes are :ref:`not thread safe <asyncio-multithreading>`.
.. versionchanged:: 3.6
The socket option ``TCP_NODELAY`` is now set by default.
BaseTransport
-------------
.. class:: BaseTransport
Base class for transports.
.. method:: close()
Close the transport. If the transport has a buffer for outgoing
data, buffered data will be flushed asynchronously. No more data
will be received. After all buffered data is flushed, the
protocol's :meth:`connection_lost` method will be called with
:const:`None` as its argument.
.. method:: is_closing()
Return ``True`` if the transport is closing or is closed.
.. versionadded:: 3.5.1
.. method:: get_extra_info(name, default=None)
Return optional transport information. *name* is a string representing
the piece of transport-specific information to get, *default* is the
value to return if the information doesn't exist.
This method allows transport implementations to easily expose
channel-specific information.
* socket:
- ``'peername'``: the remote address to which the socket is connected,
result of :meth:`socket.socket.getpeername` (``None`` on error)
- ``'socket'``: :class:`socket.socket` instance
- ``'sockname'``: the socket's own address,
result of :meth:`socket.socket.getsockname`
* SSL socket:
- ``'compression'``: the compression algorithm being used as a string,
or ``None`` if the connection isn't compressed; result of
:meth:`ssl.SSLSocket.compression`
- ``'cipher'``: a three-value tuple containing the name of the cipher
being used, the version of the SSL protocol that defines its use, and
the number of secret bits being used; result of
:meth:`ssl.SSLSocket.cipher`
- ``'peercert'``: peer certificate; result of
:meth:`ssl.SSLSocket.getpeercert`
- ``'sslcontext'``: :class:`ssl.SSLContext` instance
- ``'ssl_object'``: :class:`ssl.SSLObject` or :class:`ssl.SSLSocket`
instance
* pipe:
- ``'pipe'``: pipe object
* subprocess:
- ``'subprocess'``: :class:`subprocess.Popen` instance
.. method:: set_protocol(protocol)
Set a new protocol. Switching protocol should only be done when both
protocols are documented to support the switch.
.. versionadded:: 3.5.3
.. method:: get_protocol
Return the current protocol.
.. versionadded:: 3.5.3
.. versionchanged:: 3.5.1
``'ssl_object'`` info was added to SSL sockets.
ReadTransport
-------------
.. class:: ReadTransport
Interface for read-only transports.
.. method:: pause_reading()
Pause the receiving end of the transport. No data will be passed to
the protocol's :meth:`data_received` method until :meth:`resume_reading`
is called.
.. versionchanged:: 3.6.7
The method is idempotent, i.e. it can be called when the
transport is already paused or closed.
.. method:: resume_reading()
Resume the receiving end. The protocol's :meth:`data_received` method
will be called once again if some data is available for reading.
.. versionchanged:: 3.6.7
The method is idempotent, i.e. it can be called when the
transport is already reading.
WriteTransport
--------------
.. class:: WriteTransport
Interface for write-only transports.
.. method:: abort()
Close the transport immediately, without waiting for pending operations
to complete. Buffered data will be lost. No more data will be received.
The protocol's :meth:`connection_lost` method will eventually be
called with :const:`None` as its argument.
.. method:: can_write_eof()
Return :const:`True` if the transport supports :meth:`write_eof`,
:const:`False` if not.
.. method:: get_write_buffer_size()
Return the current size of the output buffer used by the transport.
.. method:: get_write_buffer_limits()
Get the *high*- and *low*-water limits for write flow control. Return a
tuple ``(low, high)`` where *low* and *high* are positive number of
bytes.
Use :meth:`set_write_buffer_limits` to set the limits.
.. versionadded:: 3.4.2
.. method:: set_write_buffer_limits(high=None, low=None)
Set the *high*- and *low*-water limits for write flow control.
These two values (measured in number of
bytes) control when the protocol's
:meth:`pause_writing` and :meth:`resume_writing` methods are called.
If specified, the low-water limit must be less than or equal to the
high-water limit. Neither *high* nor *low* can be negative.
:meth:`pause_writing` is called when the buffer size becomes greater
than or equal to the *high* value. If writing has been paused,
:meth:`resume_writing` is called when the buffer size becomes less
than or equal to the *low* value.
The defaults are implementation-specific. If only the
high-water limit is given, the low-water limit defaults to an
implementation-specific value less than or equal to the
high-water limit. Setting *high* to zero forces *low* to zero as
well, and causes :meth:`pause_writing` to be called whenever the
buffer becomes non-empty. Setting *low* to zero causes
:meth:`resume_writing` to be called only once the buffer is empty.
Use of zero for either limit is generally sub-optimal as it
reduces opportunities for doing I/O and computation
concurrently.
Use :meth:`get_write_buffer_limits` to get the limits.
.. method:: write(data)
Write some *data* bytes to the transport.
This method does not block; it buffers the data and arranges for it
to be sent out asynchronously.
.. method:: writelines(list_of_data)
Write a list (or any iterable) of data bytes to the transport.
This is functionally equivalent to calling :meth:`write` on each
element yielded by the iterable, but may be implemented more efficiently.
.. method:: write_eof()
Close the write end of the transport after flushing buffered data.
Data may still be received.
This method can raise :exc:`NotImplementedError` if the transport
(e.g. SSL) doesn't support half-closes.
DatagramTransport
-----------------
.. method:: DatagramTransport.sendto(data, addr=None)
Send the *data* bytes to the remote peer given by *addr* (a
transport-dependent target address). If *addr* is :const:`None`, the
data is sent to the target address given on transport creation.
This method does not block; it buffers the data and arranges for it
to be sent out asynchronously.
.. method:: DatagramTransport.abort()
Close the transport immediately, without waiting for pending operations
to complete. Buffered data will be lost. No more data will be received.
The protocol's :meth:`connection_lost` method will eventually be
called with :const:`None` as its argument.
BaseSubprocessTransport
-----------------------
.. class:: BaseSubprocessTransport
.. method:: get_pid()
Return the subprocess process id as an integer.
.. method:: get_pipe_transport(fd)
Return the transport for the communication pipe corresponding to the
integer file descriptor *fd*:
* ``0``: readable streaming transport of the standard input (*stdin*),
or :const:`None` if the subprocess was not created with ``stdin=PIPE``
* ``1``: writable streaming transport of the standard output (*stdout*),
or :const:`None` if the subprocess was not created with ``stdout=PIPE``
* ``2``: writable streaming transport of the standard error (*stderr*),
or :const:`None` if the subprocess was not created with ``stderr=PIPE``
* other *fd*: :const:`None`
.. method:: get_returncode()
Return the subprocess returncode as an integer or :const:`None`
if it hasn't returned, similarly to the
:attr:`subprocess.Popen.returncode` attribute.
.. method:: kill()
Kill the subprocess, as in :meth:`subprocess.Popen.kill`.
On POSIX systems, the function sends SIGKILL to the subprocess.
On Windows, this method is an alias for :meth:`terminate`.
.. method:: send_signal(signal)
Send the *signal* number to the subprocess, as in
:meth:`subprocess.Popen.send_signal`.
.. method:: terminate()
Ask the subprocess to stop, as in :meth:`subprocess.Popen.terminate`.
This method is an alias for the :meth:`close` method.
On POSIX systems, this method sends SIGTERM to the subprocess.
On Windows, the Windows API function TerminateProcess() is called to
stop the subprocess.
.. method:: close()
Ask the subprocess to stop by calling the :meth:`terminate` method if the
subprocess hasn't returned yet, and close transports of all pipes
(*stdin*, *stdout* and *stderr*).
.. _asyncio-protocol:
Protocols
=========
:mod:`asyncio` provides base classes that you can subclass to implement
your network protocols. Those classes are used in conjunction with
:ref:`transports <asyncio-transport>` (see below): the protocol parses incoming
data and asks for the writing of outgoing data, while the transport is
responsible for the actual I/O and buffering.
When subclassing a protocol class, it is recommended you override certain
methods. Those methods are callbacks: they will be called by the transport
on certain events (for example when some data is received); you shouldn't
call them yourself, unless you are implementing a transport.
.. note::
All callbacks have default implementations, which are empty. Therefore,
you only need to implement the callbacks for the events in which you
are interested.
Protocol classes
----------------
.. class:: Protocol
The base class for implementing streaming protocols (for use with
e.g. TCP and SSL transports).
.. class:: DatagramProtocol
The base class for implementing datagram protocols (for use with
e.g. UDP transports).
.. class:: SubprocessProtocol
The base class for implementing protocols communicating with child
processes (through a set of unidirectional pipes).
Connection callbacks
--------------------
These callbacks may be called on :class:`Protocol`, :class:`DatagramProtocol`
and :class:`SubprocessProtocol` instances:
.. method:: BaseProtocol.connection_made(transport)
Called when a connection is made.
The *transport* argument is the transport representing the
connection. You are responsible for storing it somewhere
(e.g. as an attribute) if you need to.
.. method:: BaseProtocol.connection_lost(exc)
Called when the connection is lost or closed.
The argument is either an exception object or :const:`None`.
The latter means a regular EOF is received, or the connection was
aborted or closed by this side of the connection.
:meth:`~BaseProtocol.connection_made` and :meth:`~BaseProtocol.connection_lost`
are called exactly once per successful connection. All other callbacks will be
called between those two methods, which allows for easier resource management
in your protocol implementation.
The following callbacks may be called only on :class:`SubprocessProtocol`
instances:
.. method:: SubprocessProtocol.pipe_data_received(fd, data)
Called when the child process writes data into its stdout or stderr pipe.
*fd* is the integer file descriptor of the pipe. *data* is a non-empty
bytes object containing the data.
.. method:: SubprocessProtocol.pipe_connection_lost(fd, exc)
Called when one of the pipes communicating with the child process
is closed. *fd* is the integer file descriptor that was closed.
.. method:: SubprocessProtocol.process_exited()
Called when the child process has exited.
Streaming protocols
-------------------
The following callbacks are called on :class:`Protocol` instances:
.. method:: Protocol.data_received(data)
Called when some data is received. *data* is a non-empty bytes object
containing the incoming data.
.. note::
Whether the data is buffered, chunked or reassembled depends on
the transport. In general, you shouldn't rely on specific semantics
and instead make your parsing generic and flexible enough. However,
data is always received in the correct order.
.. method:: Protocol.eof_received()
Called when the other end signals it won't send any more data
(for example by calling :meth:`write_eof`, if the other end also uses
asyncio).
This method may return a false value (including ``None``), in which case
the transport will close itself. Conversely, if this method returns a
true value, closing the transport is up to the protocol. Since the
default implementation returns ``None``, it implicitly closes the connection.
.. note::
Some transports such as SSL don't support half-closed connections,
in which case returning true from this method will not prevent closing
the connection.
:meth:`data_received` can be called an arbitrary number of times during
a connection. However, :meth:`eof_received` is called at most once
and, if called, :meth:`data_received` won't be called after it.
State machine:
start -> :meth:`~BaseProtocol.connection_made`
[-> :meth:`~Protocol.data_received` \*]
[-> :meth:`~Protocol.eof_received` ?]
-> :meth:`~BaseProtocol.connection_lost` -> end
Datagram protocols
------------------
The following callbacks are called on :class:`DatagramProtocol` instances.
.. method:: DatagramProtocol.datagram_received(data, addr)
Called when a datagram is received. *data* is a bytes object containing
the incoming data. *addr* is the address of the peer sending the data;
the exact format depends on the transport.
.. method:: DatagramProtocol.error_received(exc)
Called when a previous send or receive operation raises an
:class:`OSError`. *exc* is the :class:`OSError` instance.
This method is called in rare conditions, when the transport (e.g. UDP)
detects that a datagram couldn't be delivered to its recipient.
In many conditions though, undeliverable datagrams will be silently
dropped.
Flow control callbacks
----------------------
These callbacks may be called on :class:`Protocol`,
:class:`DatagramProtocol` and :class:`SubprocessProtocol` instances:
.. method:: BaseProtocol.pause_writing()
Called when the transport's buffer goes over the high-water mark.
.. method:: BaseProtocol.resume_writing()
Called when the transport's buffer drains below the low-water mark.
:meth:`pause_writing` and :meth:`resume_writing` calls are paired --
:meth:`pause_writing` is called once when the buffer goes strictly over
the high-water mark (even if subsequent writes increases the buffer size
even more), and eventually :meth:`resume_writing` is called once when the
buffer size reaches the low-water mark.
.. note::
If the buffer size equals the high-water mark,
:meth:`pause_writing` is not called -- it must go strictly over.
Conversely, :meth:`resume_writing` is called when the buffer size is
equal or lower than the low-water mark. These end conditions
are important to ensure that things go as expected when either
mark is zero.
.. note::
On BSD systems (OS X, FreeBSD, etc.) flow control is not supported
for :class:`DatagramProtocol`, because send failures caused by
writing too many packets cannot be detected easily. The socket
always appears 'ready' and excess packets are dropped; an
:class:`OSError` with errno set to :const:`errno.ENOBUFS` may or
may not be raised; if it is raised, it will be reported to
:meth:`DatagramProtocol.error_received` but otherwise ignored.
Coroutines and protocols
------------------------
Coroutines can be scheduled in a protocol method using :func:`ensure_future`,
but there is no guarantee made about the execution order. Protocols are not
aware of coroutines created in protocol methods and so will not wait for them.
To have a reliable execution order, use :ref:`stream objects <asyncio-streams>` in a
coroutine with ``yield from``. For example, the :meth:`StreamWriter.drain`
coroutine can be used to wait until the write buffer is flushed.
Protocol examples
=================
.. _asyncio-tcp-echo-client-protocol:
TCP echo client protocol
------------------------
TCP echo client using the :meth:`AbstractEventLoop.create_connection` method, send
data and wait until the connection is closed::
import asyncio
class EchoClientProtocol(asyncio.Protocol):
def __init__(self, message, loop):
self.message = message
self.loop = loop
def connection_made(self, transport):
transport.write(self.message.encode())
print('Data sent: {!r}'.format(self.message))
def data_received(self, data):
print('Data received: {!r}'.format(data.decode()))
def connection_lost(self, exc):
print('The server closed the connection')
print('Stop the event loop')
self.loop.stop()
loop = asyncio.get_event_loop()
message = 'Hello World!'
coro = loop.create_connection(lambda: EchoClientProtocol(message, loop),
'127.0.0.1', 8888)
loop.run_until_complete(coro)
loop.run_forever()
loop.close()
The event loop is running twice. The
:meth:`~AbstractEventLoop.run_until_complete` method is preferred in this short
example to raise an exception if the server is not listening, instead of
having to write a short coroutine to handle the exception and stop the
running loop. At :meth:`~AbstractEventLoop.run_until_complete` exit, the loop is
no longer running, so there is no need to stop the loop in case of an error.
.. seealso::
The :ref:`TCP echo client using streams <asyncio-tcp-echo-client-streams>`
example uses the :func:`asyncio.open_connection` function.
.. _asyncio-tcp-echo-server-protocol:
TCP echo server protocol
------------------------
TCP echo server using the :meth:`AbstractEventLoop.create_server` method, send back
received data and close the connection::
import asyncio
class EchoServerClientProtocol(asyncio.Protocol):
def connection_made(self, transport):
peername = transport.get_extra_info('peername')
print('Connection from {}'.format(peername))
self.transport = transport
def data_received(self, data):
message = data.decode()
print('Data received: {!r}'.format(message))
print('Send: {!r}'.format(message))
self.transport.write(data)
print('Close the client socket')
self.transport.close()
loop = asyncio.get_event_loop()
# Each client connection will create a new protocol instance
coro = loop.create_server(EchoServerClientProtocol, '127.0.0.1', 8888)
server = loop.run_until_complete(coro)
# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(server.sockets[0].getsockname()))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
# Close the server
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
:meth:`Transport.close` can be called immediately after
:meth:`WriteTransport.write` even if data are not sent yet on the socket: both
methods are asynchronous. ``yield from`` is not needed because these transport
methods are not coroutines.
.. seealso::
The :ref:`TCP echo server using streams <asyncio-tcp-echo-server-streams>`
example uses the :func:`asyncio.start_server` function.
.. _asyncio-udp-echo-client-protocol:
UDP echo client protocol
------------------------
UDP echo client using the :meth:`AbstractEventLoop.create_datagram_endpoint`
method, send data and close the transport when we received the answer::
import asyncio
class EchoClientProtocol:
def __init__(self, message, loop):
self.message = message
self.loop = loop
self.transport = None
def connection_made(self, transport):
self.transport = transport
print('Send:', self.message)
self.transport.sendto(self.message.encode())
def datagram_received(self, data, addr):
print("Received:", data.decode())
print("Close the socket")
self.transport.close()
def error_received(self, exc):
print('Error received:', exc)
def connection_lost(self, exc):
print("Socket closed, stop the event loop")
loop = asyncio.get_event_loop()
loop.stop()
loop = asyncio.get_event_loop()
message = "Hello World!"
connect = loop.create_datagram_endpoint(
lambda: EchoClientProtocol(message, loop),
remote_addr=('127.0.0.1', 9999))
transport, protocol = loop.run_until_complete(connect)
loop.run_forever()
transport.close()
loop.close()
.. _asyncio-udp-echo-server-protocol:
UDP echo server protocol
------------------------
UDP echo server using the :meth:`AbstractEventLoop.create_datagram_endpoint`
method, send back received data::
import asyncio
class EchoServerProtocol:
def connection_made(self, transport):
self.transport = transport
def datagram_received(self, data, addr):
message = data.decode()
print('Received %r from %s' % (message, addr))
print('Send %r to %s' % (message, addr))
self.transport.sendto(data, addr)
loop = asyncio.get_event_loop()
print("Starting UDP server")
# One protocol instance will be created to serve all client requests
listen = loop.create_datagram_endpoint(
EchoServerProtocol, local_addr=('127.0.0.1', 9999))
transport, protocol = loop.run_until_complete(listen)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
transport.close()
loop.close()
.. _asyncio-register-socket:
Register an open socket to wait for data using a protocol
---------------------------------------------------------
Wait until a socket receives data using the
:meth:`AbstractEventLoop.create_connection` method with a protocol, and then close
the event loop ::
import asyncio
try:
from socket import socketpair
except ImportError:
from asyncio.windows_utils import socketpair
# Create a pair of connected sockets
rsock, wsock = socketpair()
loop = asyncio.get_event_loop()
class MyProtocol(asyncio.Protocol):
transport = None
def connection_made(self, transport):
self.transport = transport
def data_received(self, data):
print("Received:", data.decode())
# We are done: close the transport (it will call connection_lost())
self.transport.close()
def connection_lost(self, exc):
# The socket has been closed, stop the event loop
loop.stop()
# Register the socket to wait for data
connect_coro = loop.create_connection(MyProtocol, sock=rsock)
transport, protocol = loop.run_until_complete(connect_coro)
# Simulate the reception of data from the network
loop.call_soon(wsock.send, 'abc'.encode())
# Run the event loop
loop.run_forever()
# We are done, close sockets and the event loop
rsock.close()
wsock.close()
loop.close()
.. seealso::
The :ref:`watch a file descriptor for read events
<asyncio-watch-read-event>` example uses the low-level
:meth:`AbstractEventLoop.add_reader` method to register the file descriptor of a
socket.
The :ref:`register an open socket to wait for data using streams
<asyncio-register-socket-streams>` example uses high-level streams
created by the :func:`open_connection` function in a coroutine.

160
third_party/python/Doc/library/asyncio-queue.rst vendored Normal file
View file

@ -0,0 +1,160 @@
.. currentmodule:: asyncio
Queues
======
**Source code:** :source:`Lib/asyncio/queues.py`
Queues:
* :class:`Queue`
* :class:`PriorityQueue`
* :class:`LifoQueue`
asyncio queue API was designed to be close to classes of the :mod:`queue`
module (:class:`~queue.Queue`, :class:`~queue.PriorityQueue`,
:class:`~queue.LifoQueue`), but it has no *timeout* parameter. The
:func:`asyncio.wait_for` function can be used to cancel a task after a timeout.
Queue
-----
.. class:: Queue(maxsize=0, \*, loop=None)
A queue, useful for coordinating producer and consumer coroutines.
If *maxsize* is less than or equal to zero, the queue size is infinite. If
it is an integer greater than ``0``, then ``yield from put()`` will block
when the queue reaches *maxsize*, until an item is removed by :meth:`get`.
Unlike the standard library :mod:`queue`, you can reliably know this Queue's
size with :meth:`qsize`, since your single-threaded asyncio application won't
be interrupted between calling :meth:`qsize` and doing an operation on the
Queue.
This class is :ref:`not thread safe <asyncio-multithreading>`.
.. versionchanged:: 3.4.4
New :meth:`join` and :meth:`task_done` methods.
.. method:: empty()
Return ``True`` if the queue is empty, ``False`` otherwise.
.. method:: full()
Return ``True`` if there are :attr:`maxsize` items in the queue.
.. note::
If the Queue was initialized with ``maxsize=0`` (the default), then
:meth:`full()` is never ``True``.
.. coroutinemethod:: get()
Remove and return an item from the queue. If queue is empty, wait until
an item is available.
This method is a :ref:`coroutine <coroutine>`.
.. seealso::
The :meth:`empty` method.
.. method:: get_nowait()
Remove and return an item from the queue.
Return an item if one is immediately available, else raise
:exc:`QueueEmpty`.
.. coroutinemethod:: join()
Block until all items in the queue have been gotten and processed.
The count of unfinished tasks goes up whenever an item is added to the
queue. The count goes down whenever a consumer thread calls
:meth:`task_done` to indicate that the item was retrieved and all work on
it is complete. When the count of unfinished tasks drops to zero,
:meth:`join` unblocks.
This method is a :ref:`coroutine <coroutine>`.
.. versionadded:: 3.4.4
.. coroutinemethod:: put(item)
Put an item into the queue. If the queue is full, wait until a free slot
is available before adding item.
This method is a :ref:`coroutine <coroutine>`.
.. seealso::
The :meth:`full` method.
.. method:: put_nowait(item)
Put an item into the queue without blocking.
If no free slot is immediately available, raise :exc:`QueueFull`.
.. method:: qsize()
Number of items in the queue.
.. method:: task_done()
Indicate that a formerly enqueued task is complete.
Used by queue consumers. For each :meth:`~Queue.get` used to fetch a task, a
subsequent call to :meth:`task_done` tells the queue that the processing
on the task is complete.
If a :meth:`join` is currently blocking, it will resume when all items
have been processed (meaning that a :meth:`task_done` call was received
for every item that had been :meth:`~Queue.put` into the queue).
Raises :exc:`ValueError` if called more times than there were items
placed in the queue.
.. versionadded:: 3.4.4
.. attribute:: maxsize
Number of items allowed in the queue.
PriorityQueue
-------------
.. class:: PriorityQueue
A subclass of :class:`Queue`; retrieves entries in priority order (lowest
first).
Entries are typically tuples of the form: (priority number, data).
LifoQueue
---------
.. class:: LifoQueue
A subclass of :class:`Queue` that retrieves most recently added entries
first.
Exceptions
^^^^^^^^^^
.. exception:: QueueEmpty
Exception raised when the :meth:`~Queue.get_nowait` method is called on a
:class:`Queue` object which is empty.
.. exception:: QueueFull
Exception raised when the :meth:`~Queue.put_nowait` method is called on a
:class:`Queue` object which is full.

471
third_party/python/Doc/library/asyncio-stream.rst vendored Normal file
View file

@ -0,0 +1,471 @@
.. currentmodule:: asyncio
.. _asyncio-streams:
+++++++++++++++++++++++++++++
Streams (coroutine based API)
+++++++++++++++++++++++++++++
**Source code:** :source:`Lib/asyncio/streams.py`
Stream functions
================
.. note::
The top-level functions in this module are meant as convenience wrappers
only; there's really nothing special there, and if they don't do
exactly what you want, feel free to copy their code.
.. coroutinefunction:: open_connection(host=None, port=None, \*, loop=None, limit=None, \*\*kwds)
A wrapper for :meth:`~AbstractEventLoop.create_connection()` returning a (reader,
writer) pair.
The reader returned is a :class:`StreamReader` instance; the writer is
a :class:`StreamWriter` instance.
The arguments are all the usual arguments to
:meth:`AbstractEventLoop.create_connection` except *protocol_factory*; most
common are positional host and port, with various optional keyword arguments
following.
Additional optional keyword arguments are *loop* (to set the event loop
instance to use) and *limit* (to set the buffer limit passed to the
:class:`StreamReader`).
This function is a :ref:`coroutine <coroutine>`.
.. coroutinefunction:: start_server(client_connected_cb, host=None, port=None, \*, loop=None, limit=None, \*\*kwds)
Start a socket server, with a callback for each client connected. The return
value is the same as :meth:`~AbstractEventLoop.create_server()`.
The *client_connected_cb* parameter is called with two parameters:
*client_reader*, *client_writer*. *client_reader* is a
:class:`StreamReader` object, while *client_writer* is a
:class:`StreamWriter` object. The *client_connected_cb* parameter can
either be a plain callback function or a :ref:`coroutine function
<coroutine>`; if it is a coroutine function, it will be automatically
converted into a :class:`Task`.
The rest of the arguments are all the usual arguments to
:meth:`~AbstractEventLoop.create_server()` except *protocol_factory*; most
common are positional *host* and *port*, with various optional keyword
arguments following.
Additional optional keyword arguments are *loop* (to set the event loop
instance to use) and *limit* (to set the buffer limit passed to the
:class:`StreamReader`).
This function is a :ref:`coroutine <coroutine>`.
.. coroutinefunction:: open_unix_connection(path=None, \*, loop=None, limit=None, **kwds)
A wrapper for :meth:`~AbstractEventLoop.create_unix_connection()` returning
a (reader, writer) pair.
See :func:`open_connection` for information about return value and other
details.
This function is a :ref:`coroutine <coroutine>`.
Availability: UNIX.
.. coroutinefunction:: start_unix_server(client_connected_cb, path=None, \*, loop=None, limit=None, **kwds)
Start a UNIX Domain Socket server, with a callback for each client connected.
See :func:`start_server` for information about return value and other
details.
This function is a :ref:`coroutine <coroutine>`.
Availability: UNIX.
StreamReader
============
.. class:: StreamReader(limit=_DEFAULT_LIMIT, loop=None)
This class is :ref:`not thread safe <asyncio-multithreading>`.
The *limit* argument's default value is set to _DEFAULT_LIMIT which is 2**16 (64 KiB)
.. method:: exception()
Get the exception.
.. method:: feed_eof()
Acknowledge the EOF.
.. method:: feed_data(data)
Feed *data* bytes in the internal buffer. Any operations waiting
for the data will be resumed.
.. method:: set_exception(exc)
Set the exception.
.. method:: set_transport(transport)
Set the transport.
.. coroutinemethod:: read(n=-1)
Read up to *n* bytes. If *n* is not provided, or set to ``-1``,
read until EOF and return all read bytes.
If the EOF was received and the internal buffer is empty,
return an empty ``bytes`` object.
This method is a :ref:`coroutine <coroutine>`.
.. coroutinemethod:: readline()
Read one line, where "line" is a sequence of bytes ending with ``\n``.
If EOF is received, and ``\n`` was not found, the method will
return the partial read bytes.
If the EOF was received and the internal buffer is empty,
return an empty ``bytes`` object.
This method is a :ref:`coroutine <coroutine>`.
.. coroutinemethod:: readexactly(n)
Read exactly *n* bytes. Raise an :exc:`IncompleteReadError` if the end of
the stream is reached before *n* can be read, the
:attr:`IncompleteReadError.partial` attribute of the exception contains
the partial read bytes.
This method is a :ref:`coroutine <coroutine>`.
.. coroutinemethod:: readuntil(separator=b'\\n')
Read data from the stream until ``separator`` is found.
On success, the data and separator will be removed from the
internal buffer (consumed). Returned data will include the
separator at the end.
Configured stream limit is used to check result. Limit sets the
maximal length of data that can be returned, not counting the
separator.
If an EOF occurs and the complete separator is still not found,
an :exc:`IncompleteReadError` exception will be
raised, and the internal buffer will be reset. The
:attr:`IncompleteReadError.partial` attribute may contain the
separator partially.
If the data cannot be read because of over limit, a
:exc:`LimitOverrunError` exception will be raised, and the data
will be left in the internal buffer, so it can be read again.
.. versionadded:: 3.5.2
.. method:: at_eof()
Return ``True`` if the buffer is empty and :meth:`feed_eof` was called.
StreamWriter
============
.. class:: StreamWriter(transport, protocol, reader, loop)
Wraps a Transport.
This exposes :meth:`write`, :meth:`writelines`, :meth:`can_write_eof()`,
:meth:`write_eof`, :meth:`get_extra_info` and :meth:`close`. It adds
:meth:`drain` which returns an optional :class:`Future` on which you can
wait for flow control. It also adds a transport attribute which references
the :class:`Transport` directly.
This class is :ref:`not thread safe <asyncio-multithreading>`.
.. attribute:: transport
Transport.
.. method:: can_write_eof()
Return :const:`True` if the transport supports :meth:`write_eof`,
:const:`False` if not. See :meth:`WriteTransport.can_write_eof`.
.. method:: close()
Close the transport: see :meth:`BaseTransport.close`.
.. coroutinemethod:: drain()
Let the write buffer of the underlying transport a chance to be flushed.
The intended use is to write::
w.write(data)
yield from w.drain()
When the size of the transport buffer reaches the high-water limit (the
protocol is paused), block until the size of the buffer is drained down
to the low-water limit and the protocol is resumed. When there is nothing
to wait for, the yield-from continues immediately.
Yielding from :meth:`drain` gives the opportunity for the loop to
schedule the write operation and flush the buffer. It should especially
be used when a possibly large amount of data is written to the transport,
and the coroutine does not yield-from between calls to :meth:`write`.
This method is a :ref:`coroutine <coroutine>`.
.. method:: get_extra_info(name, default=None)
Return optional transport information: see
:meth:`BaseTransport.get_extra_info`.
.. method:: write(data)
Write some *data* bytes to the transport: see
:meth:`WriteTransport.write`.
.. method:: writelines(data)
Write a list (or any iterable) of data bytes to the transport:
see :meth:`WriteTransport.writelines`.
.. method:: write_eof()
Close the write end of the transport after flushing buffered data:
see :meth:`WriteTransport.write_eof`.
StreamReaderProtocol
====================
.. class:: StreamReaderProtocol(stream_reader, client_connected_cb=None, loop=None)
Trivial helper class to adapt between :class:`Protocol` and
:class:`StreamReader`. Subclass of :class:`Protocol`.
*stream_reader* is a :class:`StreamReader` instance, *client_connected_cb*
is an optional function called with (stream_reader, stream_writer) when a
connection is made, *loop* is the event loop instance to use.
(This is a helper class instead of making :class:`StreamReader` itself a
:class:`Protocol` subclass, because the :class:`StreamReader` has other
potential uses, and to prevent the user of the :class:`StreamReader` from
accidentally calling inappropriate methods of the protocol.)
IncompleteReadError
===================
.. exception:: IncompleteReadError
Incomplete read error, subclass of :exc:`EOFError`.
.. attribute:: expected
Total number of expected bytes (:class:`int`).
.. attribute:: partial
Read bytes string before the end of stream was reached (:class:`bytes`).
LimitOverrunError
=================
.. exception:: LimitOverrunError
Reached the buffer limit while looking for a separator.
.. attribute:: consumed
Total number of to be consumed bytes.
Stream examples
===============
.. _asyncio-tcp-echo-client-streams:
TCP echo client using streams
-----------------------------
TCP echo client using the :func:`asyncio.open_connection` function::
import asyncio
@asyncio.coroutine
def tcp_echo_client(message, loop):
reader, writer = yield from asyncio.open_connection('127.0.0.1', 8888,
loop=loop)
print('Send: %r' % message)
writer.write(message.encode())
data = yield from reader.read(100)
print('Received: %r' % data.decode())
print('Close the socket')
writer.close()
message = 'Hello World!'
loop = asyncio.get_event_loop()
loop.run_until_complete(tcp_echo_client(message, loop))
loop.close()
.. seealso::
The :ref:`TCP echo client protocol <asyncio-tcp-echo-client-protocol>`
example uses the :meth:`AbstractEventLoop.create_connection` method.
.. _asyncio-tcp-echo-server-streams:
TCP echo server using streams
-----------------------------
TCP echo server using the :func:`asyncio.start_server` function::
import asyncio
@asyncio.coroutine
def handle_echo(reader, writer):
data = yield from reader.read(100)
message = data.decode()
addr = writer.get_extra_info('peername')
print("Received %r from %r" % (message, addr))
print("Send: %r" % message)
writer.write(data)
yield from writer.drain()
print("Close the client socket")
writer.close()
loop = asyncio.get_event_loop()
coro = asyncio.start_server(handle_echo, '127.0.0.1', 8888, loop=loop)
server = loop.run_until_complete(coro)
# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(server.sockets[0].getsockname()))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
# Close the server
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
.. seealso::
The :ref:`TCP echo server protocol <asyncio-tcp-echo-server-protocol>`
example uses the :meth:`AbstractEventLoop.create_server` method.
Get HTTP headers
----------------
Simple example querying HTTP headers of the URL passed on the command line::
import asyncio
import urllib.parse
import sys
@asyncio.coroutine
def print_http_headers(url):
url = urllib.parse.urlsplit(url)
if url.scheme == 'https':
connect = asyncio.open_connection(url.hostname, 443, ssl=True)
else:
connect = asyncio.open_connection(url.hostname, 80)
reader, writer = yield from connect
query = ('HEAD {path} HTTP/1.0\r\n'
'Host: {hostname}\r\n'
'\r\n').format(path=url.path or '/', hostname=url.hostname)
writer.write(query.encode('latin-1'))
while True:
line = yield from reader.readline()
if not line:
break
line = line.decode('latin1').rstrip()
if line:
print('HTTP header> %s' % line)
# Ignore the body, close the socket
writer.close()
url = sys.argv[1]
loop = asyncio.get_event_loop()
task = asyncio.ensure_future(print_http_headers(url))
loop.run_until_complete(task)
loop.close()
Usage::
python example.py http://example.com/path/page.html
or with HTTPS::
python example.py https://example.com/path/page.html
.. _asyncio-register-socket-streams:
Register an open socket to wait for data using streams
------------------------------------------------------
Coroutine waiting until a socket receives data using the
:func:`open_connection` function::
import asyncio
try:
from socket import socketpair
except ImportError:
from asyncio.windows_utils import socketpair
@asyncio.coroutine
def wait_for_data(loop):
# Create a pair of connected sockets
rsock, wsock = socketpair()
# Register the open socket to wait for data
reader, writer = yield from asyncio.open_connection(sock=rsock, loop=loop)
# Simulate the reception of data from the network
loop.call_soon(wsock.send, 'abc'.encode())
# Wait for data
data = yield from reader.read(100)
# Got data, we are done: close the socket
print("Received:", data.decode())
writer.close()
# Close the second socket
wsock.close()
loop = asyncio.get_event_loop()
loop.run_until_complete(wait_for_data(loop))
loop.close()
.. seealso::
The :ref:`register an open socket to wait for data using a protocol
<asyncio-register-socket>` example uses a low-level protocol created by the
:meth:`AbstractEventLoop.create_connection` method.
The :ref:`watch a file descriptor for read events
<asyncio-watch-read-event>` example uses the low-level
:meth:`AbstractEventLoop.add_reader` method to register the file descriptor of a
socket.

421
third_party/python/Doc/library/asyncio-subprocess.rst vendored Normal file
View file

@ -0,0 +1,421 @@
.. currentmodule:: asyncio
.. _asyncio-subprocess:
Subprocess
==========
**Source code:** :source:`Lib/asyncio/subprocess.py`
Windows event loop
------------------
On Windows, the default event loop is :class:`SelectorEventLoop` which does not
support subprocesses. :class:`ProactorEventLoop` should be used instead.
Example to use it on Windows::
import asyncio, sys
if sys.platform == 'win32':
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
.. seealso::
:ref:`Available event loops <asyncio-event-loops>` and :ref:`Platform
support <asyncio-platform-support>`.
Create a subprocess: high-level API using Process
-------------------------------------------------
.. coroutinefunction:: create_subprocess_exec(\*args, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds)
Create a subprocess.
The *limit* parameter sets the buffer limit passed to the
:class:`StreamReader`. See :meth:`AbstractEventLoop.subprocess_exec` for other
parameters.
Return a :class:`~asyncio.subprocess.Process` instance.
This function is a :ref:`coroutine <coroutine>`.
.. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds)
Run the shell command *cmd*.
The *limit* parameter sets the buffer limit passed to the
:class:`StreamReader`. See :meth:`AbstractEventLoop.subprocess_shell` for other
parameters.
Return a :class:`~asyncio.subprocess.Process` instance.
It is the application's responsibility to ensure that all whitespace and
metacharacters are quoted appropriately to avoid `shell injection
<https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
vulnerabilities. The :func:`shlex.quote` function can be used to properly
escape whitespace and shell metacharacters in strings that are going to be
used to construct shell commands.
This function is a :ref:`coroutine <coroutine>`.
Use the :meth:`AbstractEventLoop.connect_read_pipe` and
:meth:`AbstractEventLoop.connect_write_pipe` methods to connect pipes.
Create a subprocess: low-level API using subprocess.Popen
---------------------------------------------------------
Run subprocesses asynchronously using the :mod:`subprocess` module.
.. coroutinemethod:: AbstractEventLoop.subprocess_exec(protocol_factory, \*args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs)
Create a subprocess from one or more string arguments (character strings or
bytes strings encoded to the :ref:`filesystem encoding
<filesystem-encoding>`), where the first string
specifies the program to execute, and the remaining strings specify the
program's arguments. (Thus, together the string arguments form the
``sys.argv`` value of the program, assuming it is a Python script.) This is
similar to the standard library :class:`subprocess.Popen` class called with
shell=False and the list of strings passed as the first argument;
however, where :class:`~subprocess.Popen` takes a single argument which is
list of strings, :func:`subprocess_exec` takes multiple string arguments.
The *protocol_factory* must instantiate a subclass of the
:class:`asyncio.SubprocessProtocol` class.
Other parameters:
* *stdin*: Either a file-like object representing the pipe to be connected
to the subprocess's standard input stream using
:meth:`~AbstractEventLoop.connect_write_pipe`, or the constant
:const:`subprocess.PIPE` (the default). By default a new pipe will be
created and connected.
* *stdout*: Either a file-like object representing the pipe to be connected
to the subprocess's standard output stream using
:meth:`~AbstractEventLoop.connect_read_pipe`, or the constant
:const:`subprocess.PIPE` (the default). By default a new pipe will be
created and connected.
* *stderr*: Either a file-like object representing the pipe to be connected
to the subprocess's standard error stream using
:meth:`~AbstractEventLoop.connect_read_pipe`, or one of the constants
:const:`subprocess.PIPE` (the default) or :const:`subprocess.STDOUT`.
By default a new pipe will be created and connected. When
:const:`subprocess.STDOUT` is specified, the subprocess's standard error
stream will be connected to the same pipe as the standard output stream.
* All other keyword arguments are passed to :class:`subprocess.Popen`
without interpretation, except for *bufsize*, *universal_newlines* and
*shell*, which should not be specified at all.
Returns a pair of ``(transport, protocol)``, where *transport* is an
instance of :class:`BaseSubprocessTransport`.
This method is a :ref:`coroutine <coroutine>`.
See the constructor of the :class:`subprocess.Popen` class for parameters.
.. coroutinemethod:: AbstractEventLoop.subprocess_shell(protocol_factory, cmd, \*, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, \*\*kwargs)
Create a subprocess from *cmd*, which is a character string or a bytes
string encoded to the :ref:`filesystem encoding <filesystem-encoding>`,
using the platform's "shell" syntax. This is similar to the standard library
:class:`subprocess.Popen` class called with ``shell=True``.
The *protocol_factory* must instantiate a subclass of the
:class:`asyncio.SubprocessProtocol` class.
See :meth:`~AbstractEventLoop.subprocess_exec` for more details about
the remaining arguments.
Returns a pair of ``(transport, protocol)``, where *transport* is an
instance of :class:`BaseSubprocessTransport`.
It is the application's responsibility to ensure that all whitespace and
metacharacters are quoted appropriately to avoid `shell injection
<https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
vulnerabilities. The :func:`shlex.quote` function can be used to properly
escape whitespace and shell metacharacters in strings that are going to be
used to construct shell commands.
This method is a :ref:`coroutine <coroutine>`.
.. seealso::
The :meth:`AbstractEventLoop.connect_read_pipe` and
:meth:`AbstractEventLoop.connect_write_pipe` methods.
Constants
---------
.. data:: asyncio.subprocess.PIPE
Special value that can be used as the *stdin*, *stdout* or *stderr* argument
to :func:`create_subprocess_shell` and :func:`create_subprocess_exec` and
indicates that a pipe to the standard stream should be opened.
.. data:: asyncio.subprocess.STDOUT
Special value that can be used as the *stderr* argument to
:func:`create_subprocess_shell` and :func:`create_subprocess_exec` and
indicates that standard error should go into the same handle as standard
output.
.. data:: asyncio.subprocess.DEVNULL
Special value that can be used as the *stdin*, *stdout* or *stderr* argument
to :func:`create_subprocess_shell` and :func:`create_subprocess_exec` and
indicates that the special file :data:`os.devnull` will be used.
Process
-------
.. class:: asyncio.subprocess.Process
A subprocess created by the :func:`create_subprocess_exec` or the
:func:`create_subprocess_shell` function.
The API of the :class:`~asyncio.subprocess.Process` class was designed to be
close to the API of the :class:`subprocess.Popen` class, but there are some
differences:
* There is no explicit :meth:`~subprocess.Popen.poll` method
* The :meth:`~subprocess.Popen.communicate` and
:meth:`~subprocess.Popen.wait` methods don't take a *timeout* parameter:
use the :func:`wait_for` function
* The *universal_newlines* parameter is not supported (only bytes strings
are supported)
* The :meth:`~asyncio.subprocess.Process.wait` method of
the :class:`~asyncio.subprocess.Process` class is asynchronous whereas the
:meth:`~subprocess.Popen.wait` method of the :class:`~subprocess.Popen`
class is implemented as a busy loop.
This class is :ref:`not thread safe <asyncio-multithreading>`. See also the
:ref:`Subprocess and threads <asyncio-subprocess-threads>` section.
.. coroutinemethod:: wait()
Wait for child process to terminate. Set and return :attr:`returncode`
attribute.
This method is a :ref:`coroutine <coroutine>`.
.. note::
This will deadlock when using ``stdout=PIPE`` or ``stderr=PIPE`` and
the child process generates enough output to a pipe such that it
blocks waiting for the OS pipe buffer to accept more data. Use the
:meth:`communicate` method when using pipes to avoid that.
.. coroutinemethod:: communicate(input=None)
Interact with process: Send data to stdin. Read data from stdout and
stderr, until end-of-file is reached. Wait for process to terminate.
The optional *input* argument should be data to be sent to the child
process, or ``None``, if no data should be sent to the child. The type
of *input* must be bytes.
:meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``.
If a :exc:`BrokenPipeError` or :exc:`ConnectionResetError` exception is
raised when writing *input* into stdin, the exception is ignored. It
occurs when the process exits before all data are written into stdin.
Note that if you want to send data to the process's stdin, you need to
create the Process object with ``stdin=PIPE``. Similarly, to get anything
other than ``None`` in the result tuple, you need to give ``stdout=PIPE``
and/or ``stderr=PIPE`` too.
This method is a :ref:`coroutine <coroutine>`.
.. note::
The data read is buffered in memory, so do not use this method if the
data size is large or unlimited.
.. versionchanged:: 3.4.2
The method now ignores :exc:`BrokenPipeError` and
:exc:`ConnectionResetError`.
.. method:: send_signal(signal)
Sends the signal *signal* to the child process.
.. note::
On Windows, :py:data:`SIGTERM` is an alias for :meth:`terminate`.
``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes
started with a *creationflags* parameter which includes
``CREATE_NEW_PROCESS_GROUP``.
.. method:: terminate()
Stop the child. On Posix OSs the method sends :py:data:`signal.SIGTERM`
to the child. On Windows the Win32 API function
:c:func:`TerminateProcess` is called to stop the child.
.. method:: kill()
Kills the child. On Posix OSs the function sends :py:data:`SIGKILL` to
the child. On Windows :meth:`kill` is an alias for :meth:`terminate`.
.. attribute:: stdin
Standard input stream (:class:`StreamWriter`), ``None`` if the process
was created with ``stdin=None``.
.. attribute:: stdout
Standard output stream (:class:`StreamReader`), ``None`` if the process
was created with ``stdout=None``.
.. attribute:: stderr
Standard error stream (:class:`StreamReader`), ``None`` if the process
was created with ``stderr=None``.
.. warning::
Use the :meth:`communicate` method rather than :attr:`.stdin.write
<stdin>`, :attr:`.stdout.read <stdout>` or :attr:`.stderr.read <stderr>`
to avoid deadlocks due to streams pausing reading or writing and blocking
the child process.
.. attribute:: pid
The identifier of the process.
Note that for processes created by the :func:`create_subprocess_shell`
function, this attribute is the process identifier of the spawned shell.
.. attribute:: returncode
Return code of the process when it exited. A ``None`` value indicates
that the process has not terminated yet.
A negative value ``-N`` indicates that the child was terminated by signal
``N`` (Unix only).
.. _asyncio-subprocess-threads:
Subprocess and threads
----------------------
asyncio supports running subprocesses from different threads, but there
are limits:
* An event loop must run in the main thread
* The child watcher must be instantiated in the main thread, before executing
subprocesses from other threads. Call the :func:`get_child_watcher`
function in the main thread to instantiate the child watcher.
The :class:`asyncio.subprocess.Process` class is not thread safe.
.. seealso::
The :ref:`Concurrency and multithreading in asyncio
<asyncio-multithreading>` section.
Subprocess examples
-------------------
Subprocess using transport and protocol
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example of a subprocess protocol using to get the output of a subprocess and to
wait for the subprocess exit. The subprocess is created by the
:meth:`AbstractEventLoop.subprocess_exec` method::
import asyncio
import sys
class DateProtocol(asyncio.SubprocessProtocol):
def __init__(self, exit_future):
self.exit_future = exit_future
self.output = bytearray()
def pipe_data_received(self, fd, data):
self.output.extend(data)
def process_exited(self):
self.exit_future.set_result(True)
@asyncio.coroutine
def get_date(loop):
code = 'import datetime; print(datetime.datetime.now())'
exit_future = asyncio.Future(loop=loop)
# Create the subprocess controlled by the protocol DateProtocol,
# redirect the standard output into a pipe
create = loop.subprocess_exec(lambda: DateProtocol(exit_future),
sys.executable, '-c', code,
stdin=None, stderr=None)
transport, protocol = yield from create
# Wait for the subprocess exit using the process_exited() method
# of the protocol
yield from exit_future
# Close the stdout pipe
transport.close()
# Read the output which was collected by the pipe_data_received()
# method of the protocol
data = bytes(protocol.output)
return data.decode('ascii').rstrip()
if sys.platform == "win32":
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
else:
loop = asyncio.get_event_loop()
date = loop.run_until_complete(get_date(loop))
print("Current date: %s" % date)
loop.close()
Subprocess using streams
^^^^^^^^^^^^^^^^^^^^^^^^
Example using the :class:`~asyncio.subprocess.Process` class to control the
subprocess and the :class:`StreamReader` class to read from the standard
output. The subprocess is created by the :func:`create_subprocess_exec`
function::
import asyncio.subprocess
import sys
@asyncio.coroutine
def get_date():
code = 'import datetime; print(datetime.datetime.now())'
# Create the subprocess, redirect the standard output into a pipe
create = asyncio.create_subprocess_exec(sys.executable, '-c', code,
stdout=asyncio.subprocess.PIPE)
proc = yield from create
# Read one line of output
data = yield from proc.stdout.readline()
line = data.decode('ascii').rstrip()
# Wait for the subprocess exit
yield from proc.wait()
return line
if sys.platform == "win32":
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
else:
loop = asyncio.get_event_loop()
date = loop.run_until_complete(get_date())
print("Current date: %s" % date)
loop.close()

295
third_party/python/Doc/library/asyncio-sync.rst vendored Normal file
View file

@ -0,0 +1,295 @@
.. currentmodule:: asyncio
.. _asyncio-sync:
Synchronization primitives
==========================
**Source code:** :source:`Lib/asyncio/locks.py`
Locks:
* :class:`Lock`
* :class:`Event`
* :class:`Condition`
Semaphores:
* :class:`Semaphore`
* :class:`BoundedSemaphore`
asyncio lock API was designed to be close to classes of the :mod:`threading`
module (:class:`~threading.Lock`, :class:`~threading.Event`,
:class:`~threading.Condition`, :class:`~threading.Semaphore`,
:class:`~threading.BoundedSemaphore`), but it has no *timeout* parameter. The
:func:`asyncio.wait_for` function can be used to cancel a task after a timeout.
Locks
-----
Lock
^^^^
.. class:: Lock(\*, loop=None)
Primitive lock objects.
A primitive lock is a synchronization primitive that is not owned by a
particular coroutine when locked. A primitive lock is in one of two states,
'locked' or 'unlocked'.
It is created in the unlocked state. It has two basic methods, :meth:`acquire`
and :meth:`release`. When the state is unlocked, acquire() changes the state to
locked and returns immediately. When the state is locked, acquire() blocks
until a call to release() in another coroutine changes it to unlocked, then
the acquire() call resets it to locked and returns. The release() method
should only be called in the locked state; it changes the state to unlocked
and returns immediately. If an attempt is made to release an unlocked lock,
a :exc:`RuntimeError` will be raised.
When more than one coroutine is blocked in acquire() waiting for the state
to turn to unlocked, only one coroutine proceeds when a release() call
resets the state to unlocked; first coroutine which is blocked in acquire()
is being processed.
:meth:`acquire` is a coroutine and should be called with ``yield from``.
Locks also support the context management protocol. ``(yield from lock)``
should be used as the context manager expression.
This class is :ref:`not thread safe <asyncio-multithreading>`.
Usage::
lock = Lock()
...
yield from lock
try:
...
finally:
lock.release()
Context manager usage::
lock = Lock()
...
with (yield from lock):
...
Lock objects can be tested for locking state::
if not lock.locked():
yield from lock
else:
# lock is acquired
...
.. method:: locked()
Return ``True`` if the lock is acquired.
.. coroutinemethod:: acquire()
Acquire a lock.
This method blocks until the lock is unlocked, then sets it to locked and
returns ``True``.
This method is a :ref:`coroutine <coroutine>`.
.. method:: release()
Release a lock.
When the lock is locked, reset it to unlocked, and return. If any other
coroutines are blocked waiting for the lock to become unlocked, allow
exactly one of them to proceed.
When invoked on an unlocked lock, a :exc:`RuntimeError` is raised.
There is no return value.
Event
^^^^^
.. class:: Event(\*, loop=None)
An Event implementation, asynchronous equivalent to :class:`threading.Event`.
Class implementing event objects. An event manages a flag that can be set to
true with the :meth:`set` method and reset to false with the :meth:`clear`
method. The :meth:`wait` method blocks until the flag is true. The flag is
initially false.
This class is :ref:`not thread safe <asyncio-multithreading>`.
.. method:: clear()
Reset the internal flag to false. Subsequently, coroutines calling
:meth:`wait` will block until :meth:`set` is called to set the internal
flag to true again.
.. method:: is_set()
Return ``True`` if and only if the internal flag is true.
.. method:: set()
Set the internal flag to true. All coroutines waiting for it to become
true are awakened. Coroutine that call :meth:`wait` once the flag is true
will not block at all.
.. coroutinemethod:: wait()
Block until the internal flag is true.
If the internal flag is true on entry, return ``True`` immediately.
Otherwise, block until another coroutine calls :meth:`set` to set the
flag to true, then return ``True``.
This method is a :ref:`coroutine <coroutine>`.
Condition
^^^^^^^^^
.. class:: Condition(lock=None, \*, loop=None)
A Condition implementation, asynchronous equivalent to
:class:`threading.Condition`.
This class implements condition variable objects. A condition variable
allows one or more coroutines to wait until they are notified by another
coroutine.
If the *lock* argument is given and not ``None``, it must be a :class:`Lock`
object, and it is used as the underlying lock. Otherwise,
a new :class:`Lock` object is created and used as the underlying lock.
This class is :ref:`not thread safe <asyncio-multithreading>`.
.. coroutinemethod:: acquire()
Acquire the underlying lock.
This method blocks until the lock is unlocked, then sets it to locked and
returns ``True``.
This method is a :ref:`coroutine <coroutine>`.
.. method:: notify(n=1)
By default, wake up one coroutine waiting on this condition, if any.
If the calling coroutine has not acquired the lock when this method is
called, a :exc:`RuntimeError` is raised.
This method wakes up at most *n* of the coroutines waiting for the
condition variable; it is a no-op if no coroutines are waiting.
.. note::
An awakened coroutine does not actually return from its :meth:`wait`
call until it can reacquire the lock. Since :meth:`notify` does not
release the lock, its caller should.
.. method:: locked()
Return ``True`` if the underlying lock is acquired.
.. method:: notify_all()
Wake up all coroutines waiting on this condition. This method acts like
:meth:`notify`, but wakes up all waiting coroutines instead of one. If the
calling coroutine has not acquired the lock when this method is called, a
:exc:`RuntimeError` is raised.
.. method:: release()
Release the underlying lock.
When the lock is locked, reset it to unlocked, and return. If any other
coroutines are blocked waiting for the lock to become unlocked, allow
exactly one of them to proceed.
When invoked on an unlocked lock, a :exc:`RuntimeError` is raised.
There is no return value.
.. coroutinemethod:: wait()
Wait until notified.
If the calling coroutine has not acquired the lock when this method is
called, a :exc:`RuntimeError` is raised.
This method releases the underlying lock, and then blocks until it is
awakened by a :meth:`notify` or :meth:`notify_all` call for the same
condition variable in another coroutine. Once awakened, it re-acquires
the lock and returns ``True``.
This method is a :ref:`coroutine <coroutine>`.
.. coroutinemethod:: wait_for(predicate)
Wait until a predicate becomes true.
The predicate should be a callable which result will be interpreted as a
boolean value. The final predicate value is the return value.
This method is a :ref:`coroutine <coroutine>`.
Semaphores
----------
Semaphore
^^^^^^^^^
.. class:: Semaphore(value=1, \*, loop=None)
A Semaphore implementation.
A semaphore manages an internal counter which is decremented by each
:meth:`acquire` call and incremented by each :meth:`release` call. The
counter can never go below zero; when :meth:`acquire` finds that it is zero,
it blocks, waiting until some other coroutine calls :meth:`release`.
Semaphores also support the context management protocol.
The optional argument gives the initial value for the internal counter; it
defaults to ``1``. If the value given is less than ``0``, :exc:`ValueError`
is raised.
This class is :ref:`not thread safe <asyncio-multithreading>`.
.. coroutinemethod:: acquire()
Acquire a semaphore.
If the internal counter is larger than zero on entry, decrement it by one
and return ``True`` immediately. If it is zero on entry, block, waiting
until some other coroutine has called :meth:`release` to make it larger
than ``0``, and then return ``True``.
This method is a :ref:`coroutine <coroutine>`.
.. method:: locked()
Returns ``True`` if semaphore can not be acquired immediately.
.. method:: release()
Release a semaphore, incrementing the internal counter by one. When it
was zero on entry and another coroutine is waiting for it to become
larger than zero again, wake up that coroutine.
BoundedSemaphore
^^^^^^^^^^^^^^^^
.. class:: BoundedSemaphore(value=1, \*, loop=None)
A bounded semaphore implementation. Inherit from :class:`Semaphore`.
This raises :exc:`ValueError` in :meth:`~Semaphore.release` if it would
increase the value above the initial value.

730
third_party/python/Doc/library/asyncio-task.rst vendored Normal file
View file

@ -0,0 +1,730 @@
.. currentmodule:: asyncio
Tasks and coroutines
====================
**Source code:** :source:`Lib/asyncio/tasks.py`
**Source code:** :source:`Lib/asyncio/coroutines.py`
.. _coroutine:
Coroutines
----------
Coroutines used with :mod:`asyncio` may be implemented using the
:keyword:`async def` statement, or by using :term:`generators <generator>`.
The :keyword:`async def` type of coroutine was added in Python 3.5, and
is recommended if there is no need to support older Python versions.
Generator-based coroutines should be decorated with :func:`@asyncio.coroutine
<asyncio.coroutine>`, although this is not strictly enforced.
The decorator enables compatibility with :keyword:`async def` coroutines,
and also serves as documentation. Generator-based
coroutines use the ``yield from`` syntax introduced in :pep:`380`,
instead of the original ``yield`` syntax.
The word "coroutine", like the word "generator", is used for two
different (though related) concepts:
- The function that defines a coroutine
(a function definition using :keyword:`async def` or
decorated with ``@asyncio.coroutine``). If disambiguation is needed
we will call this a *coroutine function* (:func:`iscoroutinefunction`
returns ``True``).
- The object obtained by calling a coroutine function. This object
represents a computation or an I/O operation (usually a combination)
that will complete eventually. If disambiguation is needed we will
call it a *coroutine object* (:func:`iscoroutine` returns ``True``).
Things a coroutine can do:
- ``result = await future`` or ``result = yield from future`` --
suspends the coroutine until the
future is done, then returns the future's result, or raises an
exception, which will be propagated. (If the future is cancelled,
it will raise a ``CancelledError`` exception.) Note that tasks are
futures, and everything said about futures also applies to tasks.
- ``result = await coroutine`` or ``result = yield from coroutine`` --
wait for another coroutine to
produce a result (or raise an exception, which will be propagated).
The ``coroutine`` expression must be a *call* to another coroutine.
- ``return expression`` -- produce a result to the coroutine that is
waiting for this one using :keyword:`await` or ``yield from``.
- ``raise exception`` -- raise an exception in the coroutine that is
waiting for this one using :keyword:`await` or ``yield from``.
Calling a coroutine does not start its code running --
the coroutine object returned by the call doesn't do anything until you
schedule its execution. There are two basic ways to start it running:
call ``await coroutine`` or ``yield from coroutine`` from another coroutine
(assuming the other coroutine is already running!), or schedule its execution
using the :func:`ensure_future` function or the :meth:`AbstractEventLoop.create_task`
method.
Coroutines (and tasks) can only run when the event loop is running.
.. decorator:: coroutine
Decorator to mark generator-based coroutines. This enables
the generator use :keyword:`!yield from` to call :keyword:`async
def` coroutines, and also enables the generator to be called by
:keyword:`async def` coroutines, for instance using an
:keyword:`await` expression.
There is no need to decorate :keyword:`async def` coroutines themselves.
If the generator is not yielded from before it is destroyed, an error
message is logged. See :ref:`Detect coroutines never scheduled
<asyncio-coroutine-not-scheduled>`.
.. note::
In this documentation, some methods are documented as coroutines,
even if they are plain Python functions returning a :class:`Future`.
This is intentional to have a freedom of tweaking the implementation
of these functions in the future. If such a function is needed to be
used in a callback-style code, wrap its result with :func:`ensure_future`.
.. _asyncio-hello-world-coroutine:
Example: Hello World coroutine
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example of coroutine displaying ``"Hello World"``::
import asyncio
async def hello_world():
print("Hello World!")
loop = asyncio.get_event_loop()
# Blocking call which returns when the hello_world() coroutine is done
loop.run_until_complete(hello_world())
loop.close()
.. seealso::
The :ref:`Hello World with call_soon() <asyncio-hello-world-callback>`
example uses the :meth:`AbstractEventLoop.call_soon` method to schedule a
callback.
.. _asyncio-date-coroutine:
Example: Coroutine displaying the current date
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example of coroutine displaying the current date every second during 5 seconds
using the :meth:`sleep` function::
import asyncio
import datetime
async def display_date(loop):
end_time = loop.time() + 5.0
while True:
print(datetime.datetime.now())
if (loop.time() + 1.0) >= end_time:
break
await asyncio.sleep(1)
loop = asyncio.get_event_loop()
# Blocking call which returns when the display_date() coroutine is done
loop.run_until_complete(display_date(loop))
loop.close()
.. seealso::
The :ref:`display the current date with call_later()
<asyncio-date-callback>` example uses a callback with the
:meth:`AbstractEventLoop.call_later` method.
Example: Chain coroutines
^^^^^^^^^^^^^^^^^^^^^^^^^
Example chaining coroutines::
import asyncio
async def compute(x, y):
print("Compute %s + %s ..." % (x, y))
await asyncio.sleep(1.0)
return x + y
async def print_sum(x, y):
result = await compute(x, y)
print("%s + %s = %s" % (x, y, result))
loop = asyncio.get_event_loop()
loop.run_until_complete(print_sum(1, 2))
loop.close()
``compute()`` is chained to ``print_sum()``: ``print_sum()`` coroutine waits
until ``compute()`` is completed before returning its result.
Sequence diagram of the example:
.. image:: tulip_coro.png
:align: center
The "Task" is created by the :meth:`AbstractEventLoop.run_until_complete` method
when it gets a coroutine object instead of a task.
The diagram shows the control flow, it does not describe exactly how things
work internally. For example, the sleep coroutine creates an internal future
which uses :meth:`AbstractEventLoop.call_later` to wake up the task in 1 second.
InvalidStateError
-----------------
.. exception:: InvalidStateError
The operation is not allowed in this state.
TimeoutError
------------
.. exception:: TimeoutError
The operation exceeded the given deadline.
.. note::
This exception is different from the builtin :exc:`TimeoutError` exception!
Future
------
.. class:: Future(\*, loop=None)
This class is *almost* compatible with :class:`concurrent.futures.Future`.
Differences:
- :meth:`result` and :meth:`exception` do not take a timeout argument and
raise an exception when the future isn't done yet.
- Callbacks registered with :meth:`add_done_callback` are always called
via the event loop's :meth:`~AbstractEventLoop.call_soon`.
- This class is not compatible with the :func:`~concurrent.futures.wait` and
:func:`~concurrent.futures.as_completed` functions in the
:mod:`concurrent.futures` package.
This class is :ref:`not thread safe <asyncio-multithreading>`.
.. method:: cancel()
Cancel the future and schedule callbacks.
If the future is already done or cancelled, return ``False``. Otherwise,
change the future's state to cancelled, schedule the callbacks and return
``True``.
.. method:: cancelled()
Return ``True`` if the future was cancelled.
.. method:: done()
Return ``True`` if the future is done.
Done means either that a result / exception are available, or that the
future was cancelled.
.. method:: result()
Return the result this future represents.
If the future has been cancelled, raises :exc:`CancelledError`. If the
future's result isn't yet available, raises :exc:`InvalidStateError`. If
the future is done and has an exception set, this exception is raised.
.. method:: exception()
Return the exception that was set on this future.
The exception (or ``None`` if no exception was set) is returned only if
the future is done. If the future has been cancelled, raises
:exc:`CancelledError`. If the future isn't done yet, raises
:exc:`InvalidStateError`.
.. method:: add_done_callback(fn)
Add a callback to be run when the future becomes done.
The callback is called with a single argument - the future object. If the
future is already done when this is called, the callback is scheduled
with :meth:`~AbstractEventLoop.call_soon`.
:ref:`Use functools.partial to pass parameters to the callback
<asyncio-pass-keywords>`. For example,
``fut.add_done_callback(functools.partial(print, "Future:",
flush=True))`` will call ``print("Future:", fut, flush=True)``.
.. method:: remove_done_callback(fn)
Remove all instances of a callback from the "call when done" list.
Returns the number of callbacks removed.
.. method:: set_result(result)
Mark the future done and set its result.
If the future is already done when this method is called, raises
:exc:`InvalidStateError`.
.. method:: set_exception(exception)
Mark the future done and set an exception.
If the future is already done when this method is called, raises
:exc:`InvalidStateError`.
Example: Future with run_until_complete()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example combining a :class:`Future` and a :ref:`coroutine function
<coroutine>`::
import asyncio
async def slow_operation(future):
await asyncio.sleep(1)
future.set_result('Future is done!')
loop = asyncio.get_event_loop()
future = asyncio.Future()
asyncio.ensure_future(slow_operation(future))
loop.run_until_complete(future)
print(future.result())
loop.close()
The coroutine function is responsible for the computation (which takes 1 second)
and it stores the result into the future. The
:meth:`~AbstractEventLoop.run_until_complete` method waits for the completion of
the future.
.. note::
The :meth:`~AbstractEventLoop.run_until_complete` method uses internally the
:meth:`~Future.add_done_callback` method to be notified when the future is
done.
Example: Future with run_forever()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The previous example can be written differently using the
:meth:`Future.add_done_callback` method to describe explicitly the control
flow::
import asyncio
async def slow_operation(future):
await asyncio.sleep(1)
future.set_result('Future is done!')
def got_result(future):
print(future.result())
loop.stop()
loop = asyncio.get_event_loop()
future = asyncio.Future()
asyncio.ensure_future(slow_operation(future))
future.add_done_callback(got_result)
try:
loop.run_forever()
finally:
loop.close()
In this example, the future is used to link ``slow_operation()`` to
``got_result()``: when ``slow_operation()`` is done, ``got_result()`` is called
with the result.
Task
----
.. class:: Task(coro, \*, loop=None)
Schedule the execution of a :ref:`coroutine <coroutine>`: wrap it in a
future. A task is a subclass of :class:`Future`.
A task is responsible for executing a coroutine object in an event loop. If
the wrapped coroutine yields from a future, the task suspends the execution
of the wrapped coroutine and waits for the completion of the future. When
the future is done, the execution of the wrapped coroutine restarts with the
result or the exception of the future.
Event loops use cooperative scheduling: an event loop only runs one task at
a time. Other tasks may run in parallel if other event loops are
running in different threads. While a task waits for the completion of a
future, the event loop executes a new task.
The cancellation of a task is different from the cancelation of a
future. Calling :meth:`cancel` will throw a
:exc:`~concurrent.futures.CancelledError` to the wrapped
coroutine. :meth:`~Future.cancelled` only returns ``True`` if the
wrapped coroutine did not catch the
:exc:`~concurrent.futures.CancelledError` exception, or raised a
:exc:`~concurrent.futures.CancelledError` exception.
If a pending task is destroyed, the execution of its wrapped :ref:`coroutine
<coroutine>` did not complete. It is probably a bug and a warning is
logged: see :ref:`Pending task destroyed <asyncio-pending-task-destroyed>`.
Don't directly create :class:`Task` instances: use the :func:`ensure_future`
function or the :meth:`AbstractEventLoop.create_task` method.
This class is :ref:`not thread safe <asyncio-multithreading>`.
.. classmethod:: all_tasks(loop=None)
Return a set of all tasks for an event loop.
By default all tasks for the current event loop are returned.
.. classmethod:: current_task(loop=None)
Return the currently running task in an event loop or ``None``.
By default the current task for the current event loop is returned.
``None`` is returned when called not in the context of a :class:`Task`.
.. method:: cancel()
Request that this task cancel itself.
This arranges for a :exc:`~concurrent.futures.CancelledError` to be
thrown into the wrapped coroutine on the next cycle through the event
loop. The coroutine then has a chance to clean up or even deny the
request using try/except/finally.
Unlike :meth:`Future.cancel`, this does not guarantee that the task
will be cancelled: the exception might be caught and acted upon, delaying
cancellation of the task or preventing cancellation completely. The task
may also return a value or raise a different exception.
Immediately after this method is called, :meth:`~Future.cancelled` will
not return ``True`` (unless the task was already cancelled). A task will
be marked as cancelled when the wrapped coroutine terminates with a
:exc:`~concurrent.futures.CancelledError` exception (even if
:meth:`cancel` was not called).
.. method:: get_stack(\*, limit=None)
Return the list of stack frames for this task's coroutine.
If the coroutine is not done, this returns the stack where it is
suspended. If the coroutine has completed successfully or was
cancelled, this returns an empty list. If the coroutine was
terminated by an exception, this returns the list of traceback
frames.
The frames are always ordered from oldest to newest.
The optional limit gives the maximum number of frames to return; by
default all available frames are returned. Its meaning differs depending
on whether a stack or a traceback is returned: the newest frames of a
stack are returned, but the oldest frames of a traceback are returned.
(This matches the behavior of the traceback module.)
For reasons beyond our control, only one stack frame is returned for a
suspended coroutine.
.. method:: print_stack(\*, limit=None, file=None)
Print the stack or traceback for this task's coroutine.
This produces output similar to that of the traceback module, for the
frames retrieved by get_stack(). The limit argument is passed to
get_stack(). The file argument is an I/O stream to which the output
is written; by default output is written to sys.stderr.
Example: Parallel execution of tasks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Example executing 3 tasks (A, B, C) in parallel::
import asyncio
async def factorial(name, number):
f = 1
for i in range(2, number+1):
print("Task %s: Compute factorial(%s)..." % (name, i))
await asyncio.sleep(1)
f *= i
print("Task %s: factorial(%s) = %s" % (name, number, f))
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(
factorial("A", 2),
factorial("B", 3),
factorial("C", 4),
))
loop.close()
Output::
Task A: Compute factorial(2)...
Task B: Compute factorial(2)...
Task C: Compute factorial(2)...
Task A: factorial(2) = 2
Task B: Compute factorial(3)...
Task C: Compute factorial(3)...
Task B: factorial(3) = 6
Task C: Compute factorial(4)...
Task C: factorial(4) = 24
A task is automatically scheduled for execution when it is created. The event
loop stops when all tasks are done.
Task functions
--------------
.. note::
In the functions below, the optional *loop* argument allows explicitly setting
the event loop object used by the underlying task or coroutine. If it's
not provided, the default event loop is used.
.. function:: as_completed(fs, \*, loop=None, timeout=None)
Return an iterator whose values, when waited for, are :class:`Future`
instances.
Raises :exc:`asyncio.TimeoutError` if the timeout occurs before all Futures
are done.
Example::
for f in as_completed(fs):
result = yield from f # The 'yield from' may raise
# Use result
.. note::
The futures ``f`` are not necessarily members of fs.
.. function:: ensure_future(coro_or_future, \*, loop=None)
Schedule the execution of a :ref:`coroutine object <coroutine>`: wrap it in
a future. Return a :class:`Task` object.
If the argument is a :class:`Future`, it is returned directly.
.. versionadded:: 3.4.4
.. versionchanged:: 3.5.1
The function accepts any :term:`awaitable` object.
.. seealso::
The :meth:`AbstractEventLoop.create_task` method.
.. function:: async(coro_or_future, \*, loop=None)
A deprecated alias to :func:`ensure_future`.
.. deprecated:: 3.4.4
.. function:: wrap_future(future, \*, loop=None)
Wrap a :class:`concurrent.futures.Future` object in a :class:`Future`
object.
.. function:: gather(\*coros_or_futures, loop=None, return_exceptions=False)
Return a future aggregating results from the given coroutine objects or
futures.
All futures must share the same event loop. If all the tasks are done
successfully, the returned future's result is the list of results (in the
order of the original sequence, not necessarily the order of results
arrival). If *return_exceptions* is true, exceptions in the tasks are
treated the same as successful results, and gathered in the result list;
otherwise, the first raised exception will be immediately propagated to the
returned future.
Cancellation: if the outer Future is cancelled, all children (that have not
completed yet) are also cancelled. If any child is cancelled, this is
treated as if it raised :exc:`~concurrent.futures.CancelledError` -- the
outer Future is *not* cancelled in this case. (This is to prevent the
cancellation of one child to cause other children to be cancelled.)
.. versionchanged:: 3.6.6
If the *gather* itself is cancelled, the cancellation is propagated
regardless of *return_exceptions*.
.. function:: iscoroutine(obj)
Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`,
which may be based on a generator or an :keyword:`async def` coroutine.
.. function:: iscoroutinefunction(func)
Return ``True`` if *func* is determined to be a :ref:`coroutine function
<coroutine>`, which may be a decorated generator function or an
:keyword:`async def` function.
.. function:: run_coroutine_threadsafe(coro, loop)
Submit a :ref:`coroutine object <coroutine>` to a given event loop.
Return a :class:`concurrent.futures.Future` to access the result.
This function is meant to be called from a different thread than the one
where the event loop is running. Usage::
# Create a coroutine
coro = asyncio.sleep(1, result=3)
# Submit the coroutine to a given loop
future = asyncio.run_coroutine_threadsafe(coro, loop)
# Wait for the result with an optional timeout argument
assert future.result(timeout) == 3
If an exception is raised in the coroutine, the returned future will be
notified. It can also be used to cancel the task in the event loop::
try:
result = future.result(timeout)
except asyncio.TimeoutError:
print('The coroutine took too long, cancelling the task...')
future.cancel()
except Exception as exc:
print('The coroutine raised an exception: {!r}'.format(exc))
else:
print('The coroutine returned: {!r}'.format(result))
See the :ref:`concurrency and multithreading <asyncio-multithreading>`
section of the documentation.
.. note::
Unlike other functions from the module,
:func:`run_coroutine_threadsafe` requires the *loop* argument to
be passed explicitly.
.. versionadded:: 3.5.1
.. coroutinefunction:: sleep(delay, result=None, \*, loop=None)
Create a :ref:`coroutine <coroutine>` that completes after a given
time (in seconds). If *result* is provided, it is produced to the caller
when the coroutine completes.
The resolution of the sleep depends on the :ref:`granularity of the event
loop <asyncio-delayed-calls>`.
This function is a :ref:`coroutine <coroutine>`.
.. coroutinefunction:: shield(arg, \*, loop=None)
Wait for a future, shielding it from cancellation.
The statement::
res = yield from shield(something())
is exactly equivalent to the statement::
res = yield from something()
*except* that if the coroutine containing it is cancelled, the task running
in ``something()`` is not cancelled. From the point of view of
``something()``, the cancellation did not happen. But its caller is still
cancelled, so the yield-from expression still raises
:exc:`~concurrent.futures.CancelledError`. Note: If ``something()`` is
cancelled by other means this will still cancel ``shield()``.
If you want to completely ignore cancellation (not recommended) you can
combine ``shield()`` with a try/except clause, as follows::
try:
res = yield from shield(something())
except CancelledError:
res = None
.. coroutinefunction:: wait(futures, \*, loop=None, timeout=None,\
return_when=ALL_COMPLETED)
Wait for the Futures and coroutine objects given by the sequence *futures*
to complete. Coroutines will be wrapped in Tasks. Returns two sets of
:class:`Future`: (done, pending).
The sequence *futures* must not be empty.
*timeout* can be used to control the maximum number of seconds to wait before
returning. *timeout* can be an int or float. If *timeout* is not specified
or ``None``, there is no limit to the wait time.
*return_when* indicates when this function should return. It must be one of
the following constants of the :mod:`concurrent.futures` module:
.. tabularcolumns:: |l|L|
+-----------------------------+----------------------------------------+
| Constant | Description |
+=============================+========================================+
| :const:`FIRST_COMPLETED` | The function will return when any |
| | future finishes or is cancelled. |
+-----------------------------+----------------------------------------+
| :const:`FIRST_EXCEPTION` | The function will return when any |
| | future finishes by raising an |
| | exception. If no future raises an |
| | exception then it is equivalent to |
| | :const:`ALL_COMPLETED`. |
+-----------------------------+----------------------------------------+
| :const:`ALL_COMPLETED` | The function will return when all |
| | futures finish or are cancelled. |
+-----------------------------+----------------------------------------+
This function is a :ref:`coroutine <coroutine>`.
Usage::
done, pending = yield from asyncio.wait(fs)
.. note::
This does not raise :exc:`asyncio.TimeoutError`! Futures that aren't done
when the timeout occurs are returned in the second set.
.. coroutinefunction:: wait_for(fut, timeout, \*, loop=None)
Wait for the single :class:`Future` or :ref:`coroutine object <coroutine>`
to complete with timeout. If *timeout* is ``None``, block until the future
completes.
Coroutine will be wrapped in :class:`Task`.
Returns result of the Future or coroutine. When a timeout occurs, it
cancels the task and raises :exc:`asyncio.TimeoutError`. To avoid the task
cancellation, wrap it in :func:`shield`.
If the wait is cancelled, the future *fut* is also cancelled.
This function is a :ref:`coroutine <coroutine>`, usage::
result = yield from asyncio.wait_for(fut, 60.0)
.. versionchanged:: 3.4.3
If the wait is cancelled, the future *fut* is now also cancelled.

66
third_party/python/Doc/library/asyncio.rst vendored Normal file
View file

@ -0,0 +1,66 @@
:mod:`asyncio` --- Asynchronous I/O, event loop, coroutines and tasks
=====================================================================
.. module:: asyncio
:synopsis: Asynchronous I/O, event loop, coroutines and tasks.
.. versionadded:: 3.4
**Source code:** :source:`Lib/asyncio/`
--------------
This module provides infrastructure for writing single-threaded concurrent
code using coroutines, multiplexing I/O access over sockets and other
resources, running network clients and servers, and other related primitives.
Here is a more detailed list of the package contents:
* a pluggable :ref:`event loop <asyncio-event-loop>` with various system-specific
implementations;
* :ref:`transport <asyncio-transport>` and :ref:`protocol <asyncio-protocol>` abstractions
(similar to those in `Twisted <https://twistedmatrix.com/trac/>`_);
* concrete support for TCP, UDP, SSL, subprocess pipes, delayed calls, and
others (some may be system-dependent);
* a :class:`Future` class that mimics the one in the :mod:`concurrent.futures`
module, but adapted for use with the event loop;
* coroutines and tasks based on ``yield from`` (:PEP:`380`), to help write
concurrent code in a sequential fashion;
* cancellation support for :class:`Future`\s and coroutines;
* :ref:`synchronization primitives <asyncio-sync>` for use between coroutines in
a single thread, mimicking those in the :mod:`threading` module;
* an interface for passing work off to a threadpool, for times when
you absolutely, positively have to use a library that makes blocking
I/O calls.
Asynchronous programming is more complex than classical "sequential"
programming: see the :ref:`Develop with asyncio <asyncio-dev>` page which lists
common traps and explains how to avoid them. :ref:`Enable the debug mode
<asyncio-debug-mode>` during development to detect common issues.
Table of contents:
.. toctree::
:maxdepth: 3
asyncio-eventloop.rst
asyncio-eventloops.rst
asyncio-task.rst
asyncio-protocol.rst
asyncio-stream.rst
asyncio-subprocess.rst
asyncio-sync.rst
asyncio-queue.rst
asyncio-dev.rst
.. seealso::
The :mod:`asyncio` module was designed in :PEP:`3156`. For a
motivational primer on transports and protocols, see :PEP:`3153`.

356
third_party/python/Doc/library/asyncore.rst vendored Normal file
View file

@ -0,0 +1,356 @@
:mod:`asyncore` --- Asynchronous socket handler
===============================================
.. module:: asyncore
:synopsis: A base class for developing asynchronous socket handling
services.
.. moduleauthor:: Sam Rushing <rushing@nightmare.com>
.. sectionauthor:: Christopher Petrilli <petrilli@amber.org>
.. sectionauthor:: Steve Holden <sholden@holdenweb.com>
.. heavily adapted from original documentation by Sam Rushing
**Source code:** :source:`Lib/asyncore.py`
.. deprecated:: 3.6
Please use :mod:`asyncio` instead.
--------------
.. note::
This module exists for backwards compatibility only. For new code we
recommend using :mod:`asyncio`.
This module provides the basic infrastructure for writing asynchronous socket
service clients and servers.
There are only two ways to have a program on a single processor do "more than
one thing at a time." Multi-threaded programming is the simplest and most
popular way to do it, but there is another very different technique, that lets
you have nearly all the advantages of multi-threading, without actually using
multiple threads. It's really only practical if your program is largely I/O
bound. If your program is processor bound, then pre-emptive scheduled threads
are probably what you really need. Network servers are rarely processor
bound, however.
If your operating system supports the :c:func:`select` system call in its I/O
library (and nearly all do), then you can use it to juggle multiple
communication channels at once; doing other work while your I/O is taking
place in the "background." Although this strategy can seem strange and
complex, especially at first, it is in many ways easier to understand and
control than multi-threaded programming. The :mod:`asyncore` module solves
many of the difficult problems for you, making the task of building
sophisticated high-performance network servers and clients a snap. For
"conversational" applications and protocols the companion :mod:`asynchat`
module is invaluable.
The basic idea behind both modules is to create one or more network
*channels*, instances of class :class:`asyncore.dispatcher` and
:class:`asynchat.async_chat`. Creating the channels adds them to a global
map, used by the :func:`loop` function if you do not provide it with your own
*map*.
Once the initial channel(s) is(are) created, calling the :func:`loop` function
activates channel service, which continues until the last channel (including
any that have been added to the map during asynchronous service) is closed.
.. function:: loop([timeout[, use_poll[, map[,count]]]])
Enter a polling loop that terminates after count passes or all open
channels have been closed. All arguments are optional. The *count*
parameter defaults to ``None``, resulting in the loop terminating only when all
channels have been closed. The *timeout* argument sets the timeout
parameter for the appropriate :func:`~select.select` or :func:`~select.poll`
call, measured in seconds; the default is 30 seconds. The *use_poll*
parameter, if true, indicates that :func:`~select.poll` should be used in
preference to :func:`~select.select` (the default is ``False``).
The *map* parameter is a dictionary whose items are the channels to watch.
As channels are closed they are deleted from their map. If *map* is
omitted, a global map is used. Channels (instances of
:class:`asyncore.dispatcher`, :class:`asynchat.async_chat` and subclasses
thereof) can freely be mixed in the map.
.. class:: dispatcher()
The :class:`dispatcher` class is a thin wrapper around a low-level socket
object. To make it more useful, it has a few methods for event-handling
which are called from the asynchronous loop. Otherwise, it can be treated
as a normal non-blocking socket object.
The firing of low-level events at certain times or in certain connection
states tells the asynchronous loop that certain higher-level events have
taken place. For example, if we have asked for a socket to connect to
another host, we know that the connection has been made when the socket
becomes writable for the first time (at this point you know that you may
write to it with the expectation of success). The implied higher-level
events are:
+----------------------+----------------------------------------+
| Event | Description |
+======================+========================================+
| ``handle_connect()`` | Implied by the first read or write |
| | event |
+----------------------+----------------------------------------+
| ``handle_close()`` | Implied by a read event with no data |
| | available |
+----------------------+----------------------------------------+
| ``handle_accepted()``| Implied by a read event on a listening |
| | socket |
+----------------------+----------------------------------------+
During asynchronous processing, each mapped channel's :meth:`readable` and
:meth:`writable` methods are used to determine whether the channel's socket
should be added to the list of channels :c:func:`select`\ ed or
:c:func:`poll`\ ed for read and write events.
Thus, the set of channel events is larger than the basic socket events. The
full set of methods that can be overridden in your subclass follows:
.. method:: handle_read()
Called when the asynchronous loop detects that a :meth:`read` call on the
channel's socket will succeed.
.. method:: handle_write()
Called when the asynchronous loop detects that a writable socket can be
written. Often this method will implement the necessary buffering for
performance. For example::
def handle_write(self):
sent = self.send(self.buffer)
self.buffer = self.buffer[sent:]
.. method:: handle_expt()
Called when there is out of band (OOB) data for a socket connection. This
will almost never happen, as OOB is tenuously supported and rarely used.
.. method:: handle_connect()
Called when the active opener's socket actually makes a connection. Might
send a "welcome" banner, or initiate a protocol negotiation with the
remote endpoint, for example.
.. method:: handle_close()
Called when the socket is closed.
.. method:: handle_error()
Called when an exception is raised and not otherwise handled. The default
version prints a condensed traceback.
.. method:: handle_accept()
Called on listening channels (passive openers) when a connection can be
established with a new remote endpoint that has issued a :meth:`connect`
call for the local endpoint. Deprecated in version 3.2; use
:meth:`handle_accepted` instead.
.. deprecated:: 3.2
.. method:: handle_accepted(sock, addr)
Called on listening channels (passive openers) when a connection has been
established with a new remote endpoint that has issued a :meth:`connect`
call for the local endpoint. *sock* is a *new* socket object usable to
send and receive data on the connection, and *addr* is the address
bound to the socket on the other end of the connection.
.. versionadded:: 3.2
.. method:: readable()
Called each time around the asynchronous loop to determine whether a
channel's socket should be added to the list on which read events can
occur. The default method simply returns ``True``, indicating that by
default, all channels will be interested in read events.
.. method:: writable()
Called each time around the asynchronous loop to determine whether a
channel's socket should be added to the list on which write events can
occur. The default method simply returns ``True``, indicating that by
default, all channels will be interested in write events.
In addition, each channel delegates or extends many of the socket methods.
Most of these are nearly identical to their socket partners.
.. method:: create_socket(family=socket.AF_INET, type=socket.SOCK_STREAM)
This is identical to the creation of a normal socket, and will use the
same options for creation. Refer to the :mod:`socket` documentation for
information on creating sockets.
.. versionchanged:: 3.3
*family* and *type* arguments can be omitted.
.. method:: connect(address)
As with the normal socket object, *address* is a tuple with the first
element the host to connect to, and the second the port number.
.. method:: send(data)
Send *data* to the remote end-point of the socket.
.. method:: recv(buffer_size)
Read at most *buffer_size* bytes from the socket's remote end-point. An
empty bytes object implies that the channel has been closed from the
other end.
Note that :meth:`recv` may raise :exc:`BlockingIOError` , even though
:func:`select.select` or :func:`select.poll` has reported the socket
ready for reading.
.. method:: listen(backlog)
Listen for connections made to the socket. The *backlog* argument
specifies the maximum number of queued connections and should be at least
1; the maximum value is system-dependent (usually 5).
.. method:: bind(address)
Bind the socket to *address*. The socket must not already be bound. (The
format of *address* depends on the address family --- refer to the
:mod:`socket` documentation for more information.) To mark
the socket as re-usable (setting the :const:`SO_REUSEADDR` option), call
the :class:`dispatcher` object's :meth:`set_reuse_addr` method.
.. method:: accept()
Accept a connection. The socket must be bound to an address and listening
for connections. The return value can be either ``None`` or a pair
``(conn, address)`` where *conn* is a *new* socket object usable to send
and receive data on the connection, and *address* is the address bound to
the socket on the other end of the connection.
When ``None`` is returned it means the connection didn't take place, in
which case the server should just ignore this event and keep listening
for further incoming connections.
.. method:: close()
Close the socket. All future operations on the socket object will fail.
The remote end-point will receive no more data (after queued data is
flushed). Sockets are automatically closed when they are
garbage-collected.
.. class:: dispatcher_with_send()
A :class:`dispatcher` subclass which adds simple buffered output capability,
useful for simple clients. For more sophisticated usage use
:class:`asynchat.async_chat`.
.. class:: file_dispatcher()
A file_dispatcher takes a file descriptor or :term:`file object` along
with an optional map argument and wraps it for use with the :c:func:`poll`
or :c:func:`loop` functions. If provided a file object or anything with a
:c:func:`fileno` method, that method will be called and passed to the
:class:`file_wrapper` constructor. Availability: UNIX.
.. class:: file_wrapper()
A file_wrapper takes an integer file descriptor and calls :func:`os.dup` to
duplicate the handle so that the original handle may be closed independently
of the file_wrapper. This class implements sufficient methods to emulate a
socket for use by the :class:`file_dispatcher` class. Availability: UNIX.
.. _asyncore-example-1:
asyncore Example basic HTTP client
----------------------------------
Here is a very basic HTTP client that uses the :class:`dispatcher` class to
implement its socket handling::
import asyncore
class HTTPClient(asyncore.dispatcher):
def __init__(self, host, path):
asyncore.dispatcher.__init__(self)
self.create_socket()
self.connect( (host, 80) )
self.buffer = bytes('GET %s HTTP/1.0\r\nHost: %s\r\n\r\n' %
(path, host), 'ascii')
def handle_connect(self):
pass
def handle_close(self):
self.close()
def handle_read(self):
print(self.recv(8192))
def writable(self):
return (len(self.buffer) > 0)
def handle_write(self):
sent = self.send(self.buffer)
self.buffer = self.buffer[sent:]
client = HTTPClient('www.python.org', '/')
asyncore.loop()
.. _asyncore-example-2:
asyncore Example basic echo server
----------------------------------
Here is a basic echo server that uses the :class:`dispatcher` class to accept
connections and dispatches the incoming connections to a handler::
import asyncore
class EchoHandler(asyncore.dispatcher_with_send):
def handle_read(self):
data = self.recv(8192)
if data:
self.send(data)
class EchoServer(asyncore.dispatcher):
def __init__(self, host, port):
asyncore.dispatcher.__init__(self)
self.create_socket()
self.set_reuse_addr()
self.bind((host, port))
self.listen(5)
def handle_accepted(self, sock, addr):
print('Incoming connection from %s' % repr(addr))
handler = EchoHandler(sock)
server = EchoServer('localhost', 8080)
asyncore.loop()

109
third_party/python/Doc/library/atexit.rst vendored Normal file
View file

@ -0,0 +1,109 @@
:mod:`atexit` --- Exit handlers
===============================
.. module:: atexit
:synopsis: Register and execute cleanup functions.
.. moduleauthor:: Skip Montanaro <skip@pobox.com>
.. sectionauthor:: Skip Montanaro <skip@pobox.com>
--------------
The :mod:`atexit` module defines functions to register and unregister cleanup
functions. Functions thus registered are automatically executed upon normal
interpreter termination. :mod:`atexit` runs these functions in the *reverse*
order in which they were registered; if you register ``A``, ``B``, and ``C``,
at interpreter termination time they will be run in the order ``C``, ``B``,
``A``.
**Note:** The functions registered via this module are not called when the
program is killed by a signal not handled by Python, when a Python fatal
internal error is detected, or when :func:`os._exit` is called.
.. function:: register(func, *args, **kwargs)
Register *func* as a function to be executed at termination. Any optional
arguments that are to be passed to *func* must be passed as arguments to
:func:`register`. It is possible to register the same function and arguments
more than once.
At normal program termination (for instance, if :func:`sys.exit` is called or
the main module's execution completes), all functions registered are called in
last in, first out order. The assumption is that lower level modules will
normally be imported before higher level modules and thus must be cleaned up
later.
If an exception is raised during execution of the exit handlers, a traceback is
printed (unless :exc:`SystemExit` is raised) and the exception information is
saved. After all exit handlers have had a chance to run the last exception to
be raised is re-raised.
This function returns *func*, which makes it possible to use it as a
decorator.
.. function:: unregister(func)
Remove *func* from the list of functions to be run at interpreter
shutdown. After calling :func:`unregister`, *func* is guaranteed not to be
called when the interpreter shuts down, even if it was registered more than
once. :func:`unregister` silently does nothing if *func* was not previously
registered.
.. seealso::
Module :mod:`readline`
Useful example of :mod:`atexit` to read and write :mod:`readline` history
files.
.. _atexit-example:
:mod:`atexit` Example
---------------------
The following simple example demonstrates how a module can initialize a counter
from a file when it is imported and save the counter's updated value
automatically when the program terminates without relying on the application
making an explicit call into this module at termination. ::
try:
with open("counterfile") as infile:
_count = int(infile.read())
except FileNotFoundError:
_count = 0
def incrcounter(n):
global _count
_count = _count + n
def savecounter():
with open("counterfile", "w") as outfile:
outfile.write("%d" % _count)
import atexit
atexit.register(savecounter)
Positional and keyword arguments may also be passed to :func:`register` to be
passed along to the registered function when it is called::
def goodbye(name, adjective):
print('Goodbye, %s, it was %s to meet you.' % (name, adjective))
import atexit
atexit.register(goodbye, 'Donny', 'nice')
# or:
atexit.register(goodbye, adjective='nice', name='Donny')
Usage as a :term:`decorator`::
import atexit
@atexit.register
def goodbye():
print("You are now leaving the Python sector.")
This only works with functions that can be called without arguments.

282
third_party/python/Doc/library/audioop.rst vendored Normal file
View file

@ -0,0 +1,282 @@
:mod:`audioop` --- Manipulate raw audio data
============================================
.. module:: audioop
:synopsis: Manipulate raw audio data.
--------------
The :mod:`audioop` module contains some useful operations on sound fragments.
It operates on sound fragments consisting of signed integer samples 8, 16, 24
or 32 bits wide, stored in :term:`bytes-like objects <bytes-like object>`. All scalar items are
integers, unless specified otherwise.
.. versionchanged:: 3.4
Support for 24-bit samples was added.
All functions now accept any :term:`bytes-like object`.
String input now results in an immediate error.
.. index::
single: Intel/DVI ADPCM
single: ADPCM, Intel/DVI
single: a-LAW
single: u-LAW
This module provides support for a-LAW, u-LAW and Intel/DVI ADPCM encodings.
.. This para is mostly here to provide an excuse for the index entries...
A few of the more complicated operations only take 16-bit samples, otherwise the
sample size (in bytes) is always a parameter of the operation.
The module defines the following variables and functions:
.. exception:: error
This exception is raised on all errors, such as unknown number of bytes per
sample, etc.
.. function:: add(fragment1, fragment2, width)
Return a fragment which is the addition of the two samples passed as parameters.
*width* is the sample width in bytes, either ``1``, ``2``, ``3`` or ``4``. Both
fragments should have the same length. Samples are truncated in case of overflow.
.. function:: adpcm2lin(adpcmfragment, width, state)
Decode an Intel/DVI ADPCM coded fragment to a linear fragment. See the
description of :func:`lin2adpcm` for details on ADPCM coding. Return a tuple
``(sample, newstate)`` where the sample has the width specified in *width*.
.. function:: alaw2lin(fragment, width)
Convert sound fragments in a-LAW encoding to linearly encoded sound fragments.
a-LAW encoding always uses 8 bits samples, so *width* refers only to the sample
width of the output fragment here.
.. function:: avg(fragment, width)
Return the average over all samples in the fragment.
.. function:: avgpp(fragment, width)
Return the average peak-peak value over all samples in the fragment. No
filtering is done, so the usefulness of this routine is questionable.
.. function:: bias(fragment, width, bias)
Return a fragment that is the original fragment with a bias added to each
sample. Samples wrap around in case of overflow.
.. function:: byteswap(fragment, width)
"Byteswap" all samples in a fragment and returns the modified fragment.
Converts big-endian samples to little-endian and vice versa.
.. versionadded:: 3.4
.. function:: cross(fragment, width)
Return the number of zero crossings in the fragment passed as an argument.
.. function:: findfactor(fragment, reference)
Return a factor *F* such that ``rms(add(fragment, mul(reference, -F)))`` is
minimal, i.e., return the factor with which you should multiply *reference* to
make it match as well as possible to *fragment*. The fragments should both
contain 2-byte samples.
The time taken by this routine is proportional to ``len(fragment)``.
.. function:: findfit(fragment, reference)
Try to match *reference* as well as possible to a portion of *fragment* (which
should be the longer fragment). This is (conceptually) done by taking slices
out of *fragment*, using :func:`findfactor` to compute the best match, and
minimizing the result. The fragments should both contain 2-byte samples.
Return a tuple ``(offset, factor)`` where *offset* is the (integer) offset into
*fragment* where the optimal match started and *factor* is the (floating-point)
factor as per :func:`findfactor`.
.. function:: findmax(fragment, length)
Search *fragment* for a slice of length *length* samples (not bytes!) with
maximum energy, i.e., return *i* for which ``rms(fragment[i*2:(i+length)*2])``
is maximal. The fragments should both contain 2-byte samples.
The routine takes time proportional to ``len(fragment)``.
.. function:: getsample(fragment, width, index)
Return the value of sample *index* from the fragment.
.. function:: lin2adpcm(fragment, width, state)
Convert samples to 4 bit Intel/DVI ADPCM encoding. ADPCM coding is an adaptive
coding scheme, whereby each 4 bit number is the difference between one sample
and the next, divided by a (varying) step. The Intel/DVI ADPCM algorithm has
been selected for use by the IMA, so it may well become a standard.
*state* is a tuple containing the state of the coder. The coder returns a tuple
``(adpcmfrag, newstate)``, and the *newstate* should be passed to the next call
of :func:`lin2adpcm`. In the initial call, ``None`` can be passed as the state.
*adpcmfrag* is the ADPCM coded fragment packed 2 4-bit values per byte.
.. function:: lin2alaw(fragment, width)
Convert samples in the audio fragment to a-LAW encoding and return this as a
bytes object. a-LAW is an audio encoding format whereby you get a dynamic
range of about 13 bits using only 8 bit samples. It is used by the Sun audio
hardware, among others.
.. function:: lin2lin(fragment, width, newwidth)
Convert samples between 1-, 2-, 3- and 4-byte formats.
.. note::
In some audio formats, such as .WAV files, 16, 24 and 32 bit samples are
signed, but 8 bit samples are unsigned. So when converting to 8 bit wide
samples for these formats, you need to also add 128 to the result::
new_frames = audioop.lin2lin(frames, old_width, 1)
new_frames = audioop.bias(new_frames, 1, 128)
The same, in reverse, has to be applied when converting from 8 to 16, 24
or 32 bit width samples.
.. function:: lin2ulaw(fragment, width)
Convert samples in the audio fragment to u-LAW encoding and return this as a
bytes object. u-LAW is an audio encoding format whereby you get a dynamic
range of about 14 bits using only 8 bit samples. It is used by the Sun audio
hardware, among others.
.. function:: max(fragment, width)
Return the maximum of the *absolute value* of all samples in a fragment.
.. function:: maxpp(fragment, width)
Return the maximum peak-peak value in the sound fragment.
.. function:: minmax(fragment, width)
Return a tuple consisting of the minimum and maximum values of all samples in
the sound fragment.
.. function:: mul(fragment, width, factor)
Return a fragment that has all samples in the original fragment multiplied by
the floating-point value *factor*. Samples are truncated in case of overflow.
.. function:: ratecv(fragment, width, nchannels, inrate, outrate, state[, weightA[, weightB]])
Convert the frame rate of the input fragment.
*state* is a tuple containing the state of the converter. The converter returns
a tuple ``(newfragment, newstate)``, and *newstate* should be passed to the next
call of :func:`ratecv`. The initial call should pass ``None`` as the state.
The *weightA* and *weightB* arguments are parameters for a simple digital filter
and default to ``1`` and ``0`` respectively.
.. function:: reverse(fragment, width)
Reverse the samples in a fragment and returns the modified fragment.
.. function:: rms(fragment, width)
Return the root-mean-square of the fragment, i.e. ``sqrt(sum(S_i^2)/n)``.
This is a measure of the power in an audio signal.
.. function:: tomono(fragment, width, lfactor, rfactor)
Convert a stereo fragment to a mono fragment. The left channel is multiplied by
*lfactor* and the right channel by *rfactor* before adding the two channels to
give a mono signal.
.. function:: tostereo(fragment, width, lfactor, rfactor)
Generate a stereo fragment from a mono fragment. Each pair of samples in the
stereo fragment are computed from the mono sample, whereby left channel samples
are multiplied by *lfactor* and right channel samples by *rfactor*.
.. function:: ulaw2lin(fragment, width)
Convert sound fragments in u-LAW encoding to linearly encoded sound fragments.
u-LAW encoding always uses 8 bits samples, so *width* refers only to the sample
width of the output fragment here.
Note that operations such as :func:`.mul` or :func:`.max` make no distinction
between mono and stereo fragments, i.e. all samples are treated equal. If this
is a problem the stereo fragment should be split into two mono fragments first
and recombined later. Here is an example of how to do that::
def mul_stereo(sample, width, lfactor, rfactor):
lsample = audioop.tomono(sample, width, 1, 0)
rsample = audioop.tomono(sample, width, 0, 1)
lsample = audioop.mul(lsample, width, lfactor)
rsample = audioop.mul(rsample, width, rfactor)
lsample = audioop.tostereo(lsample, width, 1, 0)
rsample = audioop.tostereo(rsample, width, 0, 1)
return audioop.add(lsample, rsample, width)
If you use the ADPCM coder to build network packets and you want your protocol
to be stateless (i.e. to be able to tolerate packet loss) you should not only
transmit the data but also the state. Note that you should send the *initial*
state (the one you passed to :func:`lin2adpcm`) along to the decoder, not the
final state (as returned by the coder). If you want to use
:class:`struct.Struct` to store the state in binary you can code the first
element (the predicted value) in 16 bits and the second (the delta index) in 8.
The ADPCM coders have never been tried against other ADPCM coders, only against
themselves. It could well be that I misinterpreted the standards in which case
they will not be interoperable with the respective standards.
The :func:`find\*` routines might look a bit funny at first sight. They are
primarily meant to do echo cancellation. A reasonably fast way to do this is to
pick the most energetic piece of the output sample, locate that in the input
sample and subtract the whole output sample from the input sample::
def echocancel(outputdata, inputdata):
pos = audioop.findmax(outputdata, 800) # one tenth second
out_test = outputdata[pos*2:]
in_test = inputdata[pos*2:]
ipos, factor = audioop.findfit(in_test, out_test)
# Optional (for better cancellation):
# factor = audioop.findfactor(in_test[ipos*2:ipos*2+len(out_test)],
# out_test)
prefill = '\0'*(pos+ipos)*2
postfill = '\0'*(len(inputdata)-len(prefill)-len(outputdata))
outputdata = prefill + audioop.mul(outputdata, 2, -factor) + postfill
return audioop.add(inputdata, outputdata, 2)

290
third_party/python/Doc/library/base64.rst vendored Normal file
View file

@ -0,0 +1,290 @@
:mod:`base64` --- Base16, Base32, Base64, Base85 Data Encodings
===============================================================
.. module:: base64
:synopsis: RFC 3548: Base16, Base32, Base64 Data Encodings;
Base85 and Ascii85
**Source code:** :source:`Lib/base64.py`
.. index::
pair: base64; encoding
single: MIME; base64 encoding
--------------
This module provides functions for encoding binary data to printable
ASCII characters and decoding such encodings back to binary data.
It provides encoding and decoding functions for the encodings specified in
:rfc:`3548`, which defines the Base16, Base32, and Base64 algorithms,
and for the de-facto standard Ascii85 and Base85 encodings.
The :rfc:`3548` encodings are suitable for encoding binary data so that it can
safely sent by email, used as parts of URLs, or included as part of an HTTP
POST request. The encoding algorithm is not the same as the
:program:`uuencode` program.
There are two interfaces provided by this module. The modern interface
supports encoding :term:`bytes-like objects <bytes-like object>` to ASCII
:class:`bytes`, and decoding :term:`bytes-like objects <bytes-like object>` or
strings containing ASCII to :class:`bytes`. Both base-64 alphabets
defined in :rfc:`3548` (normal, and URL- and filesystem-safe) are supported.
The legacy interface does not support decoding from strings, but it does
provide functions for encoding and decoding to and from :term:`file objects
<file object>`. It only supports the Base64 standard alphabet, and it adds
newlines every 76 characters as per :rfc:`2045`. Note that if you are looking
for :rfc:`2045` support you probably want to be looking at the :mod:`email`
package instead.
.. versionchanged:: 3.3
ASCII-only Unicode strings are now accepted by the decoding functions of
the modern interface.
.. versionchanged:: 3.4
Any :term:`bytes-like objects <bytes-like object>` are now accepted by all
encoding and decoding functions in this module. Ascii85/Base85 support added.
The modern interface provides:
.. function:: b64encode(s, altchars=None)
Encode the :term:`bytes-like object` *s* using Base64 and return the encoded
:class:`bytes`.
Optional *altchars* must be a :term:`bytes-like object` of at least
length 2 (additional characters are ignored) which specifies an alternative
alphabet for the ``+`` and ``/`` characters. This allows an application to e.g.
generate URL or filesystem safe Base64 strings. The default is ``None``, for
which the standard Base64 alphabet is used.
.. function:: b64decode(s, altchars=None, validate=False)
Decode the Base64 encoded :term:`bytes-like object` or ASCII string
*s* and return the decoded :class:`bytes`.
Optional *altchars* must be a :term:`bytes-like object` or ASCII string of
at least length 2 (additional characters are ignored) which specifies the
alternative alphabet used instead of the ``+`` and ``/`` characters.
A :exc:`binascii.Error` exception is raised
if *s* is incorrectly padded.
If *validate* is ``False`` (the default), characters that are neither
in the normal base-64 alphabet nor the alternative alphabet are
discarded prior to the padding check. If *validate* is ``True``,
these non-alphabet characters in the input result in a
:exc:`binascii.Error`.
.. function:: standard_b64encode(s)
Encode :term:`bytes-like object` *s* using the standard Base64 alphabet
and return the encoded :class:`bytes`.
.. function:: standard_b64decode(s)
Decode :term:`bytes-like object` or ASCII string *s* using the standard
Base64 alphabet and return the decoded :class:`bytes`.
.. function:: urlsafe_b64encode(s)
Encode :term:`bytes-like object` *s* using the
URL- and filesystem-safe alphabet, which
substitutes ``-`` instead of ``+`` and ``_`` instead of ``/`` in the
standard Base64 alphabet, and return the encoded :class:`bytes`. The result
can still contain ``=``.
.. function:: urlsafe_b64decode(s)
Decode :term:`bytes-like object` or ASCII string *s*
using the URL- and filesystem-safe
alphabet, which substitutes ``-`` instead of ``+`` and ``_`` instead of
``/`` in the standard Base64 alphabet, and return the decoded
:class:`bytes`.
.. function:: b32encode(s)
Encode the :term:`bytes-like object` *s* using Base32 and return the
encoded :class:`bytes`.
.. function:: b32decode(s, casefold=False, map01=None)
Decode the Base32 encoded :term:`bytes-like object` or ASCII string *s* and
return the decoded :class:`bytes`.
Optional *casefold* is a flag specifying
whether a lowercase alphabet is acceptable as input. For security purposes,
the default is ``False``.
:rfc:`3548` allows for optional mapping of the digit 0 (zero) to the letter O
(oh), and for optional mapping of the digit 1 (one) to either the letter I (eye)
or letter L (el). The optional argument *map01* when not ``None``, specifies
which letter the digit 1 should be mapped to (when *map01* is not ``None``, the
digit 0 is always mapped to the letter O). For security purposes the default is
``None``, so that 0 and 1 are not allowed in the input.
A :exc:`binascii.Error` is raised if *s* is
incorrectly padded or if there are non-alphabet characters present in the
input.
.. function:: b16encode(s)
Encode the :term:`bytes-like object` *s* using Base16 and return the
encoded :class:`bytes`.
.. function:: b16decode(s, casefold=False)
Decode the Base16 encoded :term:`bytes-like object` or ASCII string *s* and
return the decoded :class:`bytes`.
Optional *casefold* is a flag specifying whether a
lowercase alphabet is acceptable as input. For security purposes, the default
is ``False``.
A :exc:`binascii.Error` is raised if *s* is
incorrectly padded or if there are non-alphabet characters present in the
input.
.. function:: a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False)
Encode the :term:`bytes-like object` *b* using Ascii85 and return the
encoded :class:`bytes`.
*foldspaces* is an optional flag that uses the special short sequence 'y'
instead of 4 consecutive spaces (ASCII 0x20) as supported by 'btoa'. This
feature is not supported by the "standard" Ascii85 encoding.
*wrapcol* controls whether the output should have newline (``b'\n'``)
characters added to it. If this is non-zero, each output line will be
at most this many characters long.
*pad* controls whether the input is padded to a multiple of 4
before encoding. Note that the ``btoa`` implementation always pads.
*adobe* controls whether the encoded byte sequence is framed with ``<~``
and ``~>``, which is used by the Adobe implementation.
.. versionadded:: 3.4
.. function:: a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \\t\\n\\r\\v')
Decode the Ascii85 encoded :term:`bytes-like object` or ASCII string *b* and
return the decoded :class:`bytes`.
*foldspaces* is a flag that specifies whether the 'y' short sequence
should be accepted as shorthand for 4 consecutive spaces (ASCII 0x20).
This feature is not supported by the "standard" Ascii85 encoding.
*adobe* controls whether the input sequence is in Adobe Ascii85 format
(i.e. is framed with <~ and ~>).
*ignorechars* should be a :term:`bytes-like object` or ASCII string
containing characters to ignore
from the input. This should only contain whitespace characters, and by
default contains all whitespace characters in ASCII.
.. versionadded:: 3.4
.. function:: b85encode(b, pad=False)
Encode the :term:`bytes-like object` *b* using base85 (as used in e.g.
git-style binary diffs) and return the encoded :class:`bytes`.
If *pad* is true, the input is padded with ``b'\0'`` so its length is a
multiple of 4 bytes before encoding.
.. versionadded:: 3.4
.. function:: b85decode(b)
Decode the base85-encoded :term:`bytes-like object` or ASCII string *b* and
return the decoded :class:`bytes`. Padding is implicitly removed, if
necessary.
.. versionadded:: 3.4
The legacy interface:
.. function:: decode(input, output)
Decode the contents of the binary *input* file and write the resulting binary
data to the *output* file. *input* and *output* must be :term:`file objects
<file object>`. *input* will be read until ``input.readline()`` returns an
empty bytes object.
.. function:: decodebytes(s)
Decode the :term:`bytes-like object` *s*, which must contain one or more
lines of base64 encoded data, and return the decoded :class:`bytes`.
.. versionadded:: 3.1
.. function:: decodestring(s)
Deprecated alias of :func:`decodebytes`.
.. deprecated:: 3.1
.. function:: encode(input, output)
Encode the contents of the binary *input* file and write the resulting base64
encoded data to the *output* file. *input* and *output* must be :term:`file
objects <file object>`. *input* will be read until ``input.read()`` returns
an empty bytes object. :func:`encode` inserts a newline character (``b'\n'``)
after every 76 bytes of the output, as well as ensuring that the output
always ends with a newline, as per :rfc:`2045` (MIME).
.. function:: encodebytes(s)
Encode the :term:`bytes-like object` *s*, which can contain arbitrary binary
data, and return :class:`bytes` containing the base64-encoded data, with newlines
(``b'\n'``) inserted after every 76 bytes of output, and ensuring that
there is a trailing newline, as per :rfc:`2045` (MIME).
.. versionadded:: 3.1
.. function:: encodestring(s)
Deprecated alias of :func:`encodebytes`.
.. deprecated:: 3.1
An example usage of the module:
>>> import base64
>>> encoded = base64.b64encode(b'data to be encoded')
>>> encoded
b'ZGF0YSB0byBiZSBlbmNvZGVk'
>>> data = base64.b64decode(encoded)
>>> data
b'data to be encoded'
.. seealso::
Module :mod:`binascii`
Support module containing ASCII-to-binary and binary-to-ASCII conversions.
:rfc:`1521` - MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies
Section 5.2, "Base64 Content-Transfer-Encoding," provides the definition of the
base64 encoding.

372
third_party/python/Doc/library/bdb.rst vendored Normal file
View file

@ -0,0 +1,372 @@
:mod:`bdb` --- Debugger framework
=================================
.. module:: bdb
:synopsis: Debugger framework.
**Source code:** :source:`Lib/bdb.py`
--------------
The :mod:`bdb` module handles basic debugger functions, like setting breakpoints
or managing execution via the debugger.
The following exception is defined:
.. exception:: BdbQuit
Exception raised by the :class:`Bdb` class for quitting the debugger.
The :mod:`bdb` module also defines two classes:
.. class:: Breakpoint(self, file, line, temporary=0, cond=None, funcname=None)
This class implements temporary breakpoints, ignore counts, disabling and
(re-)enabling, and conditionals.
Breakpoints are indexed by number through a list called :attr:`bpbynumber`
and by ``(file, line)`` pairs through :attr:`bplist`. The former points to a
single instance of class :class:`Breakpoint`. The latter points to a list of
such instances since there may be more than one breakpoint per line.
When creating a breakpoint, its associated filename should be in canonical
form. If a *funcname* is defined, a breakpoint hit will be counted when the
first line of that function is executed. A conditional breakpoint always
counts a hit.
:class:`Breakpoint` instances have the following methods:
.. method:: deleteMe()
Delete the breakpoint from the list associated to a file/line. If it is
the last breakpoint in that position, it also deletes the entry for the
file/line.
.. method:: enable()
Mark the breakpoint as enabled.
.. method:: disable()
Mark the breakpoint as disabled.
.. method:: bpformat()
Return a string with all the information about the breakpoint, nicely
formatted:
* The breakpoint number.
* If it is temporary or not.
* Its file,line position.
* The condition that causes a break.
* If it must be ignored the next N times.
* The breakpoint hit count.
.. versionadded:: 3.2
.. method:: bpprint(out=None)
Print the output of :meth:`bpformat` to the file *out*, or if it is
``None``, to standard output.
.. class:: Bdb(skip=None)
The :class:`Bdb` class acts as a generic Python debugger base class.
This class takes care of the details of the trace facility; a derived class
should implement user interaction. The standard debugger class
(:class:`pdb.Pdb`) is an example.
The *skip* argument, if given, must be an iterable of glob-style
module name patterns. The debugger will not step into frames that
originate in a module that matches one of these patterns. Whether a
frame is considered to originate in a certain module is determined
by the ``__name__`` in the frame globals.
.. versionadded:: 3.1
The *skip* argument.
The following methods of :class:`Bdb` normally don't need to be overridden.
.. method:: canonic(filename)
Auxiliary method for getting a filename in a canonical form, that is, as a
case-normalized (on case-insensitive filesystems) absolute path, stripped
of surrounding angle brackets.
.. method:: reset()
Set the :attr:`botframe`, :attr:`stopframe`, :attr:`returnframe` and
:attr:`quitting` attributes with values ready to start debugging.
.. method:: trace_dispatch(frame, event, arg)
This function is installed as the trace function of debugged frames. Its
return value is the new trace function (in most cases, that is, itself).
The default implementation decides how to dispatch a frame, depending on
the type of event (passed as a string) that is about to be executed.
*event* can be one of the following:
* ``"line"``: A new line of code is going to be executed.
* ``"call"``: A function is about to be called, or another code block
entered.
* ``"return"``: A function or other code block is about to return.
* ``"exception"``: An exception has occurred.
* ``"c_call"``: A C function is about to be called.
* ``"c_return"``: A C function has returned.
* ``"c_exception"``: A C function has raised an exception.
For the Python events, specialized functions (see below) are called. For
the C events, no action is taken.
The *arg* parameter depends on the previous event.
See the documentation for :func:`sys.settrace` for more information on the
trace function. For more information on code and frame objects, refer to
:ref:`types`.
.. method:: dispatch_line(frame)
If the debugger should stop on the current line, invoke the
:meth:`user_line` method (which should be overridden in subclasses).
Raise a :exc:`BdbQuit` exception if the :attr:`Bdb.quitting` flag is set
(which can be set from :meth:`user_line`). Return a reference to the
:meth:`trace_dispatch` method for further tracing in that scope.
.. method:: dispatch_call(frame, arg)
If the debugger should stop on this function call, invoke the
:meth:`user_call` method (which should be overridden in subclasses).
Raise a :exc:`BdbQuit` exception if the :attr:`Bdb.quitting` flag is set
(which can be set from :meth:`user_call`). Return a reference to the
:meth:`trace_dispatch` method for further tracing in that scope.
.. method:: dispatch_return(frame, arg)
If the debugger should stop on this function return, invoke the
:meth:`user_return` method (which should be overridden in subclasses).
Raise a :exc:`BdbQuit` exception if the :attr:`Bdb.quitting` flag is set
(which can be set from :meth:`user_return`). Return a reference to the
:meth:`trace_dispatch` method for further tracing in that scope.
.. method:: dispatch_exception(frame, arg)
If the debugger should stop at this exception, invokes the
:meth:`user_exception` method (which should be overridden in subclasses).
Raise a :exc:`BdbQuit` exception if the :attr:`Bdb.quitting` flag is set
(which can be set from :meth:`user_exception`). Return a reference to the
:meth:`trace_dispatch` method for further tracing in that scope.
Normally derived classes don't override the following methods, but they may
if they want to redefine the definition of stopping and breakpoints.
.. method:: stop_here(frame)
This method checks if the *frame* is somewhere below :attr:`botframe` in
the call stack. :attr:`botframe` is the frame in which debugging started.
.. method:: break_here(frame)
This method checks if there is a breakpoint in the filename and line
belonging to *frame* or, at least, in the current function. If the
breakpoint is a temporary one, this method deletes it.
.. method:: break_anywhere(frame)
This method checks if there is a breakpoint in the filename of the current
frame.
Derived classes should override these methods to gain control over debugger
operation.
.. method:: user_call(frame, argument_list)
This method is called from :meth:`dispatch_call` when there is the
possibility that a break might be necessary anywhere inside the called
function.
.. method:: user_line(frame)
This method is called from :meth:`dispatch_line` when either
:meth:`stop_here` or :meth:`break_here` yields ``True``.
.. method:: user_return(frame, return_value)
This method is called from :meth:`dispatch_return` when :meth:`stop_here`
yields ``True``.
.. method:: user_exception(frame, exc_info)
This method is called from :meth:`dispatch_exception` when
:meth:`stop_here` yields ``True``.
.. method:: do_clear(arg)
Handle how a breakpoint must be removed when it is a temporary one.
This method must be implemented by derived classes.
Derived classes and clients can call the following methods to affect the
stepping state.
.. method:: set_step()
Stop after one line of code.
.. method:: set_next(frame)
Stop on the next line in or below the given frame.
.. method:: set_return(frame)
Stop when returning from the given frame.
.. method:: set_until(frame)
Stop when the line with the line no greater than the current one is
reached or when returning from current frame.
.. method:: set_trace([frame])
Start debugging from *frame*. If *frame* is not specified, debugging
starts from caller's frame.
.. method:: set_continue()
Stop only at breakpoints or when finished. If there are no breakpoints,
set the system trace function to ``None``.
.. method:: set_quit()
Set the :attr:`quitting` attribute to ``True``. This raises :exc:`BdbQuit` in
the next call to one of the :meth:`dispatch_\*` methods.
Derived classes and clients can call the following methods to manipulate
breakpoints. These methods return a string containing an error message if
something went wrong, or ``None`` if all is well.
.. method:: set_break(filename, lineno, temporary=0, cond, funcname)
Set a new breakpoint. If the *lineno* line doesn't exist for the
*filename* passed as argument, return an error message. The *filename*
should be in canonical form, as described in the :meth:`canonic` method.
.. method:: clear_break(filename, lineno)
Delete the breakpoints in *filename* and *lineno*. If none were set, an
error message is returned.
.. method:: clear_bpbynumber(arg)
Delete the breakpoint which has the index *arg* in the
:attr:`Breakpoint.bpbynumber`. If *arg* is not numeric or out of range,
return an error message.
.. method:: clear_all_file_breaks(filename)
Delete all breakpoints in *filename*. If none were set, an error message
is returned.
.. method:: clear_all_breaks()
Delete all existing breakpoints.
.. method:: get_bpbynumber(arg)
Return a breakpoint specified by the given number. If *arg* is a string,
it will be converted to a number. If *arg* is a non-numeric string, if
the given breakpoint never existed or has been deleted, a
:exc:`ValueError` is raised.
.. versionadded:: 3.2
.. method:: get_break(filename, lineno)
Check if there is a breakpoint for *lineno* of *filename*.
.. method:: get_breaks(filename, lineno)
Return all breakpoints for *lineno* in *filename*, or an empty list if
none are set.
.. method:: get_file_breaks(filename)
Return all breakpoints in *filename*, or an empty list if none are set.
.. method:: get_all_breaks()
Return all breakpoints that are set.
Derived classes and clients can call the following methods to get a data
structure representing a stack trace.
.. method:: get_stack(f, t)
Get a list of records for a frame and all higher (calling) and lower
frames, and the size of the higher part.
.. method:: format_stack_entry(frame_lineno, lprefix=': ')
Return a string with information about a stack entry, identified by a
``(frame, lineno)`` tuple:
* The canonical form of the filename which contains the frame.
* The function name, or ``"<lambda>"``.
* The input arguments.
* The return value.
* The line of code (if it exists).
The following two methods can be called by clients to use a debugger to debug
a :term:`statement`, given as a string.
.. method:: run(cmd, globals=None, locals=None)
Debug a statement executed via the :func:`exec` function. *globals*
defaults to :attr:`__main__.__dict__`, *locals* defaults to *globals*.
.. method:: runeval(expr, globals=None, locals=None)
Debug an expression executed via the :func:`eval` function. *globals* and
*locals* have the same meaning as in :meth:`run`.
.. method:: runctx(cmd, globals, locals)
For backwards compatibility. Calls the :meth:`run` method.
.. method:: runcall(func, *args, **kwds)
Debug a single function call, and return its result.
Finally, the module defines the following functions:
.. function:: checkfuncname(b, frame)
Check whether we should break here, depending on the way the breakpoint *b*
was set.
If it was set via line number, it checks if ``b.line`` is the same as the one
in the frame also passed as argument. If the breakpoint was set via function
name, we have to check we are in the right frame (the right function) and if
we are in its first executable line.
.. function:: effective(file, line, frame)
Determine if there is an effective (active) breakpoint at this line of code.
Return a tuple of the breakpoint and a boolean that indicates if it is ok
to delete a temporary breakpoint. Return ``(None, None)`` if there is no
matching breakpoint.
.. function:: set_trace()
Start debugging with a :class:`Bdb` instance from caller's frame.

23
third_party/python/Doc/library/binary.rst vendored Normal file
View file

@ -0,0 +1,23 @@
.. _binaryservices:
********************
Binary Data Services
********************
The modules described in this chapter provide some basic services operations
for manipulation of binary data. Other operations on binary data, specifically
in relation to file formats and network protocols, are described in the
relevant sections.
Some libraries described under :ref:`textservices` also work with either
ASCII-compatible binary formats (for example, :mod:`re`) or all binary data
(for example, :mod:`difflib`).
In addition, see the documentation for Python's built-in binary data types in
:ref:`binaryseq`.
.. toctree::
struct.rst
codecs.rst

186
third_party/python/Doc/library/binascii.rst vendored Normal file
View file

@ -0,0 +1,186 @@
:mod:`binascii` --- Convert between binary and ASCII
====================================================
.. module:: binascii
:synopsis: Tools for converting between binary and various ASCII-encoded binary
representations.
.. index::
module: uu
module: base64
module: binhex
--------------
The :mod:`binascii` module contains a number of methods to convert between
binary and various ASCII-encoded binary representations. Normally, you will not
use these functions directly but use wrapper modules like :mod:`uu`,
:mod:`base64`, or :mod:`binhex` instead. The :mod:`binascii` module contains
low-level functions written in C for greater speed that are used by the
higher-level modules.
.. note::
``a2b_*`` functions accept Unicode strings containing only ASCII characters.
Other functions only accept :term:`bytes-like objects <bytes-like object>` (such as
:class:`bytes`, :class:`bytearray` and other objects that support the buffer
protocol).
.. versionchanged:: 3.3
ASCII-only unicode strings are now accepted by the ``a2b_*`` functions.
The :mod:`binascii` module defines the following functions:
.. function:: a2b_uu(string)
Convert a single line of uuencoded data back to binary and return the binary
data. Lines normally contain 45 (binary) bytes, except for the last line. Line
data may be followed by whitespace.
.. function:: b2a_uu(data)
Convert binary data to a line of ASCII characters, the return value is the
converted line, including a newline char. The length of *data* should be at most
45.
.. function:: a2b_base64(string)
Convert a block of base64 data back to binary and return the binary data. More
than one line may be passed at a time.
.. function:: b2a_base64(data, \*, newline=True)
Convert binary data to a line of ASCII characters in base64 coding. The return
value is the converted line, including a newline char if *newline* is
true. The output of this function conforms to :rfc:`3548`.
.. versionchanged:: 3.6
Added the *newline* parameter.
.. function:: a2b_qp(data, header=False)
Convert a block of quoted-printable data back to binary and return the binary
data. More than one line may be passed at a time. If the optional argument
*header* is present and true, underscores will be decoded as spaces.
.. function:: b2a_qp(data, quotetabs=False, istext=True, header=False)
Convert binary data to a line(s) of ASCII characters in quoted-printable
encoding. The return value is the converted line(s). If the optional argument
*quotetabs* is present and true, all tabs and spaces will be encoded. If the
optional argument *istext* is present and true, newlines are not encoded but
trailing whitespace will be encoded. If the optional argument *header* is
present and true, spaces will be encoded as underscores per :rfc:`1522`. If the
optional argument *header* is present and false, newline characters will be
encoded as well; otherwise linefeed conversion might corrupt the binary data
stream.
.. function:: a2b_hqx(string)
Convert binhex4 formatted ASCII data to binary, without doing RLE-decompression.
The string should contain a complete number of binary bytes, or (in case of the
last portion of the binhex4 data) have the remaining bits zero.
.. function:: rledecode_hqx(data)
Perform RLE-decompression on the data, as per the binhex4 standard. The
algorithm uses ``0x90`` after a byte as a repeat indicator, followed by a count.
A count of ``0`` specifies a byte value of ``0x90``. The routine returns the
decompressed data, unless data input data ends in an orphaned repeat indicator,
in which case the :exc:`Incomplete` exception is raised.
.. versionchanged:: 3.2
Accept only bytestring or bytearray objects as input.
.. function:: rlecode_hqx(data)
Perform binhex4 style RLE-compression on *data* and return the result.
.. function:: b2a_hqx(data)
Perform hexbin4 binary-to-ASCII translation and return the resulting string. The
argument should already be RLE-coded, and have a length divisible by 3 (except
possibly the last fragment).
.. function:: crc_hqx(data, value)
Compute a 16-bit CRC value of *data*, starting with *value* as the
initial CRC, and return the result. This uses the CRC-CCITT polynomial
*x*:sup:`16` + *x*:sup:`12` + *x*:sup:`5` + 1, often represented as
0x1021. This CRC is used in the binhex4 format.
.. function:: crc32(data[, value])
Compute CRC-32, the 32-bit checksum of *data*, starting with an
initial CRC of *value*. The default initial CRC is zero. The algorithm
is consistent with the ZIP file checksum. Since the algorithm is designed for
use as a checksum algorithm, it is not suitable for use as a general hash
algorithm. Use as follows::
print(binascii.crc32(b"hello world"))
# Or, in two pieces:
crc = binascii.crc32(b"hello")
crc = binascii.crc32(b" world", crc)
print('crc32 = {:#010x}'.format(crc))
.. versionchanged:: 3.0
The result is always unsigned.
To generate the same numeric value across all Python versions and
platforms, use ``crc32(data) & 0xffffffff``.
.. function:: b2a_hex(data)
hexlify(data)
Return the hexadecimal representation of the binary *data*. Every byte of
*data* is converted into the corresponding 2-digit hex representation. The
returned bytes object is therefore twice as long as the length of *data*.
.. function:: a2b_hex(hexstr)
unhexlify(hexstr)
Return the binary data represented by the hexadecimal string *hexstr*. This
function is the inverse of :func:`b2a_hex`. *hexstr* must contain an even number
of hexadecimal digits (which can be upper or lower case), otherwise an
:exc:`Error` exception is raised.
.. exception:: Error
Exception raised on errors. These are usually programming errors.
.. exception:: Incomplete
Exception raised on incomplete data. These are usually not programming errors,
but may be handled by reading a little more data and trying again.
.. seealso::
Module :mod:`base64`
Support for RFC compliant base64-style encoding in base 16, 32, 64,
and 85.
Module :mod:`binhex`
Support for the binhex format used on the Macintosh.
Module :mod:`uu`
Support for UU encoding used on Unix.
Module :mod:`quopri`
Support for quoted-printable encoding used in MIME email messages.

57
third_party/python/Doc/library/binhex.rst vendored Normal file
View file

@ -0,0 +1,57 @@
:mod:`binhex` --- Encode and decode binhex4 files
=================================================
.. module:: binhex
:synopsis: Encode and decode files in binhex4 format.
**Source code:** :source:`Lib/binhex.py`
--------------
This module encodes and decodes files in binhex4 format, a format allowing
representation of Macintosh files in ASCII. Only the data fork is handled.
The :mod:`binhex` module defines the following functions:
.. function:: binhex(input, output)
Convert a binary file with filename *input* to binhex file *output*. The
*output* parameter can either be a filename or a file-like object (any object
supporting a :meth:`write` and :meth:`close` method).
.. function:: hexbin(input, output)
Decode a binhex file *input*. *input* may be a filename or a file-like object
supporting :meth:`read` and :meth:`close` methods. The resulting file is written
to a file named *output*, unless the argument is ``None`` in which case the
output filename is read from the binhex file.
The following exception is also defined:
.. exception:: Error
Exception raised when something can't be encoded using the binhex format (for
example, a filename is too long to fit in the filename field), or when input is
not properly encoded binhex data.
.. seealso::
Module :mod:`binascii`
Support module containing ASCII-to-binary and binary-to-ASCII conversions.
.. _binhex-notes:
Notes
-----
There is an alternative, more powerful interface to the coder and decoder, see
the source for details.
If you code or decode textfiles on non-Macintosh platforms they will still use
the old Macintosh newline convention (carriage-return as end of line).

149
third_party/python/Doc/library/bisect.rst vendored Normal file
View file

@ -0,0 +1,149 @@
:mod:`bisect` --- Array bisection algorithm
===========================================
.. module:: bisect
:synopsis: Array bisection algorithms for binary searching.
.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
.. sectionauthor:: Raymond Hettinger <python at rcn.com>
.. example based on the PyModules FAQ entry by Aaron Watters <arw@pythonpros.com>
**Source code:** :source:`Lib/bisect.py`
--------------
This module provides support for maintaining a list in sorted order without
having to sort the list after each insertion. For long lists of items with
expensive comparison operations, this can be an improvement over the more common
approach. The module is called :mod:`bisect` because it uses a basic bisection
algorithm to do its work. The source code may be most useful as a working
example of the algorithm (the boundary conditions are already right!).
The following functions are provided:
.. function:: bisect_left(a, x, lo=0, hi=len(a))
Locate the insertion point for *x* in *a* to maintain sorted order.
The parameters *lo* and *hi* may be used to specify a subset of the list
which should be considered; by default the entire list is used. If *x* is
already present in *a*, the insertion point will be before (to the left of)
any existing entries. The return value is suitable for use as the first
parameter to ``list.insert()`` assuming that *a* is already sorted.
The returned insertion point *i* partitions the array *a* into two halves so
that ``all(val < x for val in a[lo:i])`` for the left side and
``all(val >= x for val in a[i:hi])`` for the right side.
.. function:: bisect_right(a, x, lo=0, hi=len(a))
bisect(a, x, lo=0, hi=len(a))
Similar to :func:`bisect_left`, but returns an insertion point which comes
after (to the right of) any existing entries of *x* in *a*.
The returned insertion point *i* partitions the array *a* into two halves so
that ``all(val <= x for val in a[lo:i])`` for the left side and
``all(val > x for val in a[i:hi])`` for the right side.
.. function:: insort_left(a, x, lo=0, hi=len(a))
Insert *x* in *a* in sorted order. This is equivalent to
``a.insert(bisect.bisect_left(a, x, lo, hi), x)`` assuming that *a* is
already sorted. Keep in mind that the O(log n) search is dominated by
the slow O(n) insertion step.
.. function:: insort_right(a, x, lo=0, hi=len(a))
insort(a, x, lo=0, hi=len(a))
Similar to :func:`insort_left`, but inserting *x* in *a* after any existing
entries of *x*.
.. seealso::
`SortedCollection recipe
<https://code.activestate.com/recipes/577197-sortedcollection/>`_ that uses
bisect to build a full-featured collection class with straight-forward search
methods and support for a key-function. The keys are precomputed to save
unnecessary calls to the key function during searches.
Searching Sorted Lists
----------------------
The above :func:`bisect` functions are useful for finding insertion points but
can be tricky or awkward to use for common searching tasks. The following five
functions show how to transform them into the standard lookups for sorted
lists::
def index(a, x):
'Locate the leftmost value exactly equal to x'
i = bisect_left(a, x)
if i != len(a) and a[i] == x:
return i
raise ValueError
def find_lt(a, x):
'Find rightmost value less than x'
i = bisect_left(a, x)
if i:
return a[i-1]
raise ValueError
def find_le(a, x):
'Find rightmost value less than or equal to x'
i = bisect_right(a, x)
if i:
return a[i-1]
raise ValueError
def find_gt(a, x):
'Find leftmost value greater than x'
i = bisect_right(a, x)
if i != len(a):
return a[i]
raise ValueError
def find_ge(a, x):
'Find leftmost item greater than or equal to x'
i = bisect_left(a, x)
if i != len(a):
return a[i]
raise ValueError
Other Examples
--------------
.. _bisect-example:
The :func:`bisect` function can be useful for numeric table lookups. This
example uses :func:`bisect` to look up a letter grade for an exam score (say)
based on a set of ordered numeric breakpoints: 90 and up is an 'A', 80 to 89 is
a 'B', and so on::
>>> def grade(score, breakpoints=[60, 70, 80, 90], grades='FDCBA'):
... i = bisect(breakpoints, score)
... return grades[i]
...
>>> [grade(score) for score in [33, 99, 77, 70, 89, 90, 100]]
['F', 'A', 'C', 'C', 'B', 'A', 'A']
Unlike the :func:`sorted` function, it does not make sense for the :func:`bisect`
functions to have *key* or *reversed* arguments because that would lead to an
inefficient design (successive calls to bisect functions would not "remember"
all of the previous key lookups).
Instead, it is better to search a list of precomputed keys to find the index
of the record in question::
>>> data = [('red', 5), ('blue', 1), ('yellow', 8), ('black', 0)]
>>> data.sort(key=lambda r: r[1])
>>> keys = [r[1] for r in data] # precomputed list of keys
>>> data[bisect_left(keys, 0)]
('black', 0)
>>> data[bisect_left(keys, 1)]
('blue', 1)
>>> data[bisect_left(keys, 5)]
('red', 5)
>>> data[bisect_left(keys, 8)]
('yellow', 8)

42
third_party/python/Doc/library/builtins.rst vendored Normal file
View file

@ -0,0 +1,42 @@
:mod:`builtins` --- Built-in objects
====================================
.. module:: builtins
:synopsis: The module that provides the built-in namespace.
--------------
This module provides direct access to all 'built-in' identifiers of Python; for
example, ``builtins.open`` is the full name for the built-in function
:func:`open`. See :ref:`built-in-funcs` and :ref:`built-in-consts` for
documentation.
This module is not normally accessed explicitly by most applications, but can be
useful in modules that provide objects with the same name as a built-in value,
but in which the built-in of that name is also needed. For example, in a module
that wants to implement an :func:`open` function that wraps the built-in
:func:`open`, this module can be used directly::
import builtins
def open(path):
f = builtins.open(path, 'r')
return UpperCaser(f)
class UpperCaser:
'''Wrapper around a file that converts output to upper-case.'''
def __init__(self, f):
self._f = f
def read(self, count=-1):
return self._f.read(count).upper()
# ...
As an implementation detail, most modules have the name ``__builtins__`` made
available as part of their globals. The value of ``__builtins__`` is normally
either this module or the value of this module's :attr:`~object.__dict__` attribute.
Since this is an implementation detail, it may not be used by alternate
implementations of Python.

252
third_party/python/Doc/library/bz2.rst vendored Normal file
View file

@ -0,0 +1,252 @@
:mod:`bz2` --- Support for :program:`bzip2` compression
=======================================================
.. module:: bz2
:synopsis: Interfaces for bzip2 compression and decompression.
.. moduleauthor:: Gustavo Niemeyer <niemeyer@conectiva.com>
.. moduleauthor:: Nadeem Vawda <nadeem.vawda@gmail.com>
.. sectionauthor:: Gustavo Niemeyer <niemeyer@conectiva.com>
.. sectionauthor:: Nadeem Vawda <nadeem.vawda@gmail.com>
**Source code:** :source:`Lib/bz2.py`
--------------
This module provides a comprehensive interface for compressing and
decompressing data using the bzip2 compression algorithm.
The :mod:`bz2` module contains:
* The :func:`.open` function and :class:`BZ2File` class for reading and
writing compressed files.
* The :class:`BZ2Compressor` and :class:`BZ2Decompressor` classes for
incremental (de)compression.
* The :func:`compress` and :func:`decompress` functions for one-shot
(de)compression.
All of the classes in this module may safely be accessed from multiple threads.
(De)compression of files
------------------------
.. function:: open(filename, mode='r', compresslevel=9, encoding=None, errors=None, newline=None)
Open a bzip2-compressed file in binary or text mode, returning a :term:`file
object`.
As with the constructor for :class:`BZ2File`, the *filename* argument can be
an actual filename (a :class:`str` or :class:`bytes` object), or an existing
file object to read from or write to.
The *mode* argument can be any of ``'r'``, ``'rb'``, ``'w'``, ``'wb'``,
``'x'``, ``'xb'``, ``'a'`` or ``'ab'`` for binary mode, or ``'rt'``,
``'wt'``, ``'xt'``, or ``'at'`` for text mode. The default is ``'rb'``.
The *compresslevel* argument is an integer from 1 to 9, as for the
:class:`BZ2File` constructor.
For binary mode, this function is equivalent to the :class:`BZ2File`
constructor: ``BZ2File(filename, mode, compresslevel=compresslevel)``. In
this case, the *encoding*, *errors* and *newline* arguments must not be
provided.
For text mode, a :class:`BZ2File` object is created, and wrapped in an
:class:`io.TextIOWrapper` instance with the specified encoding, error
handling behavior, and line ending(s).
.. versionadded:: 3.3
.. versionchanged:: 3.4
The ``'x'`` (exclusive creation) mode was added.
.. versionchanged:: 3.6
Accepts a :term:`path-like object`.
.. class:: BZ2File(filename, mode='r', buffering=None, compresslevel=9)
Open a bzip2-compressed file in binary mode.
If *filename* is a :class:`str` or :class:`bytes` object, open the named file
directly. Otherwise, *filename* should be a :term:`file object`, which will
be used to read or write the compressed data.
The *mode* argument can be either ``'r'`` for reading (default), ``'w'`` for
overwriting, ``'x'`` for exclusive creation, or ``'a'`` for appending. These
can equivalently be given as ``'rb'``, ``'wb'``, ``'xb'`` and ``'ab'``
respectively.
If *filename* is a file object (rather than an actual file name), a mode of
``'w'`` does not truncate the file, and is instead equivalent to ``'a'``.
The *buffering* argument is ignored. Its use is deprecated.
If *mode* is ``'w'`` or ``'a'``, *compresslevel* can be a number between
``1`` and ``9`` specifying the level of compression: ``1`` produces the
least compression, and ``9`` (default) produces the most compression.
If *mode* is ``'r'``, the input file may be the concatenation of multiple
compressed streams.
:class:`BZ2File` provides all of the members specified by the
:class:`io.BufferedIOBase`, except for :meth:`detach` and :meth:`truncate`.
Iteration and the :keyword:`with` statement are supported.
:class:`BZ2File` also provides the following method:
.. method:: peek([n])
Return buffered data without advancing the file position. At least one
byte of data will be returned (unless at EOF). The exact number of bytes
returned is unspecified.
.. note:: While calling :meth:`peek` does not change the file position of
the :class:`BZ2File`, it may change the position of the underlying file
object (e.g. if the :class:`BZ2File` was constructed by passing a file
object for *filename*).
.. versionadded:: 3.3
.. versionchanged:: 3.1
Support for the :keyword:`with` statement was added.
.. versionchanged:: 3.3
The :meth:`fileno`, :meth:`readable`, :meth:`seekable`, :meth:`writable`,
:meth:`read1` and :meth:`readinto` methods were added.
.. versionchanged:: 3.3
Support was added for *filename* being a :term:`file object` instead of an
actual filename.
.. versionchanged:: 3.3
The ``'a'`` (append) mode was added, along with support for reading
multi-stream files.
.. versionchanged:: 3.4
The ``'x'`` (exclusive creation) mode was added.
.. versionchanged:: 3.5
The :meth:`~io.BufferedIOBase.read` method now accepts an argument of
``None``.
.. versionchanged:: 3.6
Accepts a :term:`path-like object`.
Incremental (de)compression
---------------------------
.. class:: BZ2Compressor(compresslevel=9)
Create a new compressor object. This object may be used to compress data
incrementally. For one-shot compression, use the :func:`compress` function
instead.
*compresslevel*, if given, must be a number between ``1`` and ``9``. The
default is ``9``.
.. method:: compress(data)
Provide data to the compressor object. Returns a chunk of compressed data
if possible, or an empty byte string otherwise.
When you have finished providing data to the compressor, call the
:meth:`flush` method to finish the compression process.
.. method:: flush()
Finish the compression process. Returns the compressed data left in
internal buffers.
The compressor object may not be used after this method has been called.
.. class:: BZ2Decompressor()
Create a new decompressor object. This object may be used to decompress data
incrementally. For one-shot compression, use the :func:`decompress` function
instead.
.. note::
This class does not transparently handle inputs containing multiple
compressed streams, unlike :func:`decompress` and :class:`BZ2File`. If
you need to decompress a multi-stream input with :class:`BZ2Decompressor`,
you must use a new decompressor for each stream.
.. method:: decompress(data, max_length=-1)
Decompress *data* (a :term:`bytes-like object`), returning
uncompressed data as bytes. Some of *data* may be buffered
internally, for use in later calls to :meth:`decompress`. The
returned data should be concatenated with the output of any
previous calls to :meth:`decompress`.
If *max_length* is nonnegative, returns at most *max_length*
bytes of decompressed data. If this limit is reached and further
output can be produced, the :attr:`~.needs_input` attribute will
be set to ``False``. In this case, the next call to
:meth:`~.decompress` may provide *data* as ``b''`` to obtain
more of the output.
If all of the input data was decompressed and returned (either
because this was less than *max_length* bytes, or because
*max_length* was negative), the :attr:`~.needs_input` attribute
will be set to ``True``.
Attempting to decompress data after the end of stream is reached
raises an `EOFError`. Any data found after the end of the
stream is ignored and saved in the :attr:`~.unused_data` attribute.
.. versionchanged:: 3.5
Added the *max_length* parameter.
.. attribute:: eof
``True`` if the end-of-stream marker has been reached.
.. versionadded:: 3.3
.. attribute:: unused_data
Data found after the end of the compressed stream.
If this attribute is accessed before the end of the stream has been
reached, its value will be ``b''``.
.. attribute:: needs_input
``False`` if the :meth:`.decompress` method can provide more
decompressed data before requiring new uncompressed input.
.. versionadded:: 3.5
One-shot (de)compression
------------------------
.. function:: compress(data, compresslevel=9)
Compress *data*.
*compresslevel*, if given, must be a number between ``1`` and ``9``. The
default is ``9``.
For incremental compression, use a :class:`BZ2Compressor` instead.
.. function:: decompress(data)
Decompress *data*.
If *data* is the concatenation of multiple compressed streams, decompress
all of the streams.
For incremental decompression, use a :class:`BZ2Decompressor` instead.
.. versionchanged:: 3.3
Support for multi-stream inputs was added.

316
third_party/python/Doc/library/calendar.rst vendored Normal file
View file

@ -0,0 +1,316 @@
:mod:`calendar` --- General calendar-related functions
======================================================
.. module:: calendar
:synopsis: Functions for working with calendars, including some emulation
of the Unix cal program.
.. sectionauthor:: Drew Csillag <drew_csillag@geocities.com>
**Source code:** :source:`Lib/calendar.py`
--------------
This module allows you to output calendars like the Unix :program:`cal` program,
and provides additional useful functions related to the calendar. By default,
these calendars have Monday as the first day of the week, and Sunday as the last
(the European convention). Use :func:`setfirstweekday` to set the first day of
the week to Sunday (6) or to any other weekday. Parameters that specify dates
are given as integers. For related
functionality, see also the :mod:`datetime` and :mod:`time` modules.
Most of these functions and classes rely on the :mod:`datetime` module which
uses an idealized calendar, the current Gregorian calendar extended
in both directions. This matches the definition of the "proleptic Gregorian"
calendar in Dershowitz and Reingold's book "Calendrical Calculations", where
it's the base calendar for all computations.
.. class:: Calendar(firstweekday=0)
Creates a :class:`Calendar` object. *firstweekday* is an integer specifying the
first day of the week. ``0`` is Monday (the default), ``6`` is Sunday.
A :class:`Calendar` object provides several methods that can be used for
preparing the calendar data for formatting. This class doesn't do any formatting
itself. This is the job of subclasses.
:class:`Calendar` instances have the following methods:
.. method:: iterweekdays()
Return an iterator for the week day numbers that will be used for one
week. The first value from the iterator will be the same as the value of
the :attr:`firstweekday` property.
.. method:: itermonthdates(year, month)
Return an iterator for the month *month* (1--12) in the year *year*. This
iterator will return all days (as :class:`datetime.date` objects) for the
month and all days before the start of the month or after the end of the
month that are required to get a complete week.
.. method:: itermonthdays2(year, month)
Return an iterator for the month *month* in the year *year* similar to
:meth:`itermonthdates`. Days returned will be tuples consisting of a day
number and a week day number.
.. method:: itermonthdays(year, month)
Return an iterator for the month *month* in the year *year* similar to
:meth:`itermonthdates`. Days returned will simply be day numbers.
.. method:: monthdatescalendar(year, month)
Return a list of the weeks in the month *month* of the *year* as full
weeks. Weeks are lists of seven :class:`datetime.date` objects.
.. method:: monthdays2calendar(year, month)
Return a list of the weeks in the month *month* of the *year* as full
weeks. Weeks are lists of seven tuples of day numbers and weekday
numbers.
.. method:: monthdayscalendar(year, month)
Return a list of the weeks in the month *month* of the *year* as full
weeks. Weeks are lists of seven day numbers.
.. method:: yeardatescalendar(year, width=3)
Return the data for the specified year ready for formatting. The return
value is a list of month rows. Each month row contains up to *width*
months (defaulting to 3). Each month contains between 4 and 6 weeks and
each week contains 1--7 days. Days are :class:`datetime.date` objects.
.. method:: yeardays2calendar(year, width=3)
Return the data for the specified year ready for formatting (similar to
:meth:`yeardatescalendar`). Entries in the week lists are tuples of day
numbers and weekday numbers. Day numbers outside this month are zero.
.. method:: yeardayscalendar(year, width=3)
Return the data for the specified year ready for formatting (similar to
:meth:`yeardatescalendar`). Entries in the week lists are day numbers. Day
numbers outside this month are zero.
.. class:: TextCalendar(firstweekday=0)
This class can be used to generate plain text calendars.
:class:`TextCalendar` instances have the following methods:
.. method:: formatmonth(theyear, themonth, w=0, l=0)
Return a month's calendar in a multi-line string. If *w* is provided, it
specifies the width of the date columns, which are centered. If *l* is
given, it specifies the number of lines that each week will use. Depends
on the first weekday as specified in the constructor or set by the
:meth:`setfirstweekday` method.
.. method:: prmonth(theyear, themonth, w=0, l=0)
Print a month's calendar as returned by :meth:`formatmonth`.
.. method:: formatyear(theyear, w=2, l=1, c=6, m=3)
Return a *m*-column calendar for an entire year as a multi-line string.
Optional parameters *w*, *l*, and *c* are for date column width, lines per
week, and number of spaces between month columns, respectively. Depends on
the first weekday as specified in the constructor or set by the
:meth:`setfirstweekday` method. The earliest year for which a calendar
can be generated is platform-dependent.
.. method:: pryear(theyear, w=2, l=1, c=6, m=3)
Print the calendar for an entire year as returned by :meth:`formatyear`.
.. class:: HTMLCalendar(firstweekday=0)
This class can be used to generate HTML calendars.
:class:`HTMLCalendar` instances have the following methods:
.. method:: formatmonth(theyear, themonth, withyear=True)
Return a month's calendar as an HTML table. If *withyear* is true the year
will be included in the header, otherwise just the month name will be
used.
.. method:: formatyear(theyear, width=3)
Return a year's calendar as an HTML table. *width* (defaulting to 3)
specifies the number of months per row.
.. method:: formatyearpage(theyear, width=3, css='calendar.css', encoding=None)
Return a year's calendar as a complete HTML page. *width* (defaulting to
3) specifies the number of months per row. *css* is the name for the
cascading style sheet to be used. :const:`None` can be passed if no style
sheet should be used. *encoding* specifies the encoding to be used for the
output (defaulting to the system default encoding).
.. class:: LocaleTextCalendar(firstweekday=0, locale=None)
This subclass of :class:`TextCalendar` can be passed a locale name in the
constructor and will return month and weekday names in the specified locale.
If this locale includes an encoding all strings containing month and weekday
names will be returned as unicode.
.. class:: LocaleHTMLCalendar(firstweekday=0, locale=None)
This subclass of :class:`HTMLCalendar` can be passed a locale name in the
constructor and will return month and weekday names in the specified
locale. If this locale includes an encoding all strings containing month and
weekday names will be returned as unicode.
.. note::
The :meth:`formatweekday` and :meth:`formatmonthname` methods of these two
classes temporarily change the current locale to the given *locale*. Because
the current locale is a process-wide setting, they are not thread-safe.
For simple text calendars this module provides the following functions.
.. function:: setfirstweekday(weekday)
Sets the weekday (``0`` is Monday, ``6`` is Sunday) to start each week. The
values :const:`MONDAY`, :const:`TUESDAY`, :const:`WEDNESDAY`, :const:`THURSDAY`,
:const:`FRIDAY`, :const:`SATURDAY`, and :const:`SUNDAY` are provided for
convenience. For example, to set the first weekday to Sunday::
import calendar
calendar.setfirstweekday(calendar.SUNDAY)
.. function:: firstweekday()
Returns the current setting for the weekday to start each week.
.. function:: isleap(year)
Returns :const:`True` if *year* is a leap year, otherwise :const:`False`.
.. function:: leapdays(y1, y2)
Returns the number of leap years in the range from *y1* to *y2* (exclusive),
where *y1* and *y2* are years.
This function works for ranges spanning a century change.
.. function:: weekday(year, month, day)
Returns the day of the week (``0`` is Monday) for *year* (``1970``--...),
*month* (``1``--``12``), *day* (``1``--``31``).
.. function:: weekheader(n)
Return a header containing abbreviated weekday names. *n* specifies the width in
characters for one weekday.
.. function:: monthrange(year, month)
Returns weekday of first day of the month and number of days in month, for the
specified *year* and *month*.
.. function:: monthcalendar(year, month)
Returns a matrix representing a month's calendar. Each row represents a week;
days outside of the month a represented by zeros. Each week begins with Monday
unless set by :func:`setfirstweekday`.
.. function:: prmonth(theyear, themonth, w=0, l=0)
Prints a month's calendar as returned by :func:`month`.
.. function:: month(theyear, themonth, w=0, l=0)
Returns a month's calendar in a multi-line string using the :meth:`formatmonth`
of the :class:`TextCalendar` class.
.. function:: prcal(year, w=0, l=0, c=6, m=3)
Prints the calendar for an entire year as returned by :func:`calendar`.
.. function:: calendar(year, w=2, l=1, c=6, m=3)
Returns a 3-column calendar for an entire year as a multi-line string using
the :meth:`formatyear` of the :class:`TextCalendar` class.
.. function:: timegm(tuple)
An unrelated but handy function that takes a time tuple such as returned by
the :func:`~time.gmtime` function in the :mod:`time` module, and returns the
corresponding Unix timestamp value, assuming an epoch of 1970, and the POSIX
encoding. In fact, :func:`time.gmtime` and :func:`timegm` are each others'
inverse.
The :mod:`calendar` module exports the following data attributes:
.. data:: day_name
An array that represents the days of the week in the current locale.
.. data:: day_abbr
An array that represents the abbreviated days of the week in the current locale.
.. data:: month_name
An array that represents the months of the year in the current locale. This
follows normal convention of January being month number 1, so it has a length of
13 and ``month_name[0]`` is the empty string.
.. data:: month_abbr
An array that represents the abbreviated months of the year in the current
locale. This follows normal convention of January being month number 1, so it
has a length of 13 and ``month_abbr[0]`` is the empty string.
.. seealso::
Module :mod:`datetime`
Object-oriented interface to dates and times with similar functionality to the
:mod:`time` module.
Module :mod:`time`
Low-level time related functions.

540
third_party/python/Doc/library/cgi.rst vendored Normal file
View file

@ -0,0 +1,540 @@
:mod:`cgi` --- Common Gateway Interface support
===============================================
.. module:: cgi
:synopsis: Helpers for running Python scripts via the Common Gateway Interface.
**Source code:** :source:`Lib/cgi.py`
.. index::
pair: WWW; server
pair: CGI; protocol
pair: HTTP; protocol
pair: MIME; headers
single: URL
single: Common Gateway Interface
--------------
Support module for Common Gateway Interface (CGI) scripts.
This module defines a number of utilities for use by CGI scripts written in
Python.
Introduction
------------
.. _cgi-intro:
A CGI script is invoked by an HTTP server, usually to process user input
submitted through an HTML ``<FORM>`` or ``<ISINDEX>`` element.
Most often, CGI scripts live in the server's special :file:`cgi-bin` directory.
The HTTP server places all sorts of information about the request (such as the
client's hostname, the requested URL, the query string, and lots of other
goodies) in the script's shell environment, executes the script, and sends the
script's output back to the client.
The script's input is connected to the client too, and sometimes the form data
is read this way; at other times the form data is passed via the "query string"
part of the URL. This module is intended to take care of the different cases
and provide a simpler interface to the Python script. It also provides a number
of utilities that help in debugging scripts, and the latest addition is support
for file uploads from a form (if your browser supports it).
The output of a CGI script should consist of two sections, separated by a blank
line. The first section contains a number of headers, telling the client what
kind of data is following. Python code to generate a minimal header section
looks like this::
print("Content-Type: text/html") # HTML is following
print() # blank line, end of headers
The second section is usually HTML, which allows the client software to display
nicely formatted text with header, in-line images, etc. Here's Python code that
prints a simple piece of HTML::
print("<TITLE>CGI script output</TITLE>")
print("<H1>This is my first CGI script</H1>")
print("Hello, world!")
.. _using-the-cgi-module:
Using the cgi module
--------------------
Begin by writing ``import cgi``.
When you write a new script, consider adding these lines::
import cgitb
cgitb.enable()
This activates a special exception handler that will display detailed reports in
the Web browser if any errors occur. If you'd rather not show the guts of your
program to users of your script, you can have the reports saved to files
instead, with code like this::
import cgitb
cgitb.enable(display=0, logdir="/path/to/logdir")
It's very helpful to use this feature during script development. The reports
produced by :mod:`cgitb` provide information that can save you a lot of time in
tracking down bugs. You can always remove the ``cgitb`` line later when you
have tested your script and are confident that it works correctly.
To get at submitted form data, use the :class:`FieldStorage` class. If the form
contains non-ASCII characters, use the *encoding* keyword parameter set to the
value of the encoding defined for the document. It is usually contained in the
META tag in the HEAD section of the HTML document or by the
:mailheader:`Content-Type` header). This reads the form contents from the
standard input or the environment (depending on the value of various
environment variables set according to the CGI standard). Since it may consume
standard input, it should be instantiated only once.
The :class:`FieldStorage` instance can be indexed like a Python dictionary.
It allows membership testing with the :keyword:`in` operator, and also supports
the standard dictionary method :meth:`~dict.keys` and the built-in function
:func:`len`. Form fields containing empty strings are ignored and do not appear
in the dictionary; to keep such values, provide a true value for the optional
*keep_blank_values* keyword parameter when creating the :class:`FieldStorage`
instance.
For instance, the following code (which assumes that the
:mailheader:`Content-Type` header and blank line have already been printed)
checks that the fields ``name`` and ``addr`` are both set to a non-empty
string::
form = cgi.FieldStorage()
if "name" not in form or "addr" not in form:
print("<H1>Error</H1>")
print("Please fill in the name and addr fields.")
return
print("<p>name:", form["name"].value)
print("<p>addr:", form["addr"].value)
...further form processing here...
Here the fields, accessed through ``form[key]``, are themselves instances of
:class:`FieldStorage` (or :class:`MiniFieldStorage`, depending on the form
encoding). The :attr:`~FieldStorage.value` attribute of the instance yields
the string value of the field. The :meth:`~FieldStorage.getvalue` method
returns this string value directly; it also accepts an optional second argument
as a default to return if the requested key is not present.
If the submitted form data contains more than one field with the same name, the
object retrieved by ``form[key]`` is not a :class:`FieldStorage` or
:class:`MiniFieldStorage` instance but a list of such instances. Similarly, in
this situation, ``form.getvalue(key)`` would return a list of strings. If you
expect this possibility (when your HTML form contains multiple fields with the
same name), use the :meth:`~FieldStorage.getlist` method, which always returns
a list of values (so that you do not need to special-case the single item
case). For example, this code concatenates any number of username fields,
separated by commas::
value = form.getlist("username")
usernames = ",".join(value)
If a field represents an uploaded file, accessing the value via the
:attr:`~FieldStorage.value` attribute or the :meth:`~FieldStorage.getvalue`
method reads the entire file in memory as bytes. This may not be what you
want. You can test for an uploaded file by testing either the
:attr:`~FieldStorage.filename` attribute or the :attr:`~FieldStorage.file`
attribute. You can then read the data from the :attr:`!file`
attribute before it is automatically closed as part of the garbage collection of
the :class:`FieldStorage` instance
(the :func:`~io.RawIOBase.read` and :func:`~io.IOBase.readline` methods will
return bytes)::
fileitem = form["userfile"]
if fileitem.file:
# It's an uploaded file; count lines
linecount = 0
while True:
line = fileitem.file.readline()
if not line: break
linecount = linecount + 1
:class:`FieldStorage` objects also support being used in a :keyword:`with`
statement, which will automatically close them when done.
If an error is encountered when obtaining the contents of an uploaded file
(for example, when the user interrupts the form submission by clicking on
a Back or Cancel button) the :attr:`~FieldStorage.done` attribute of the
object for the field will be set to the value -1.
The file upload draft standard entertains the possibility of uploading multiple
files from one field (using a recursive :mimetype:`multipart/\*` encoding).
When this occurs, the item will be a dictionary-like :class:`FieldStorage` item.
This can be determined by testing its :attr:`!type` attribute, which should be
:mimetype:`multipart/form-data` (or perhaps another MIME type matching
:mimetype:`multipart/\*`). In this case, it can be iterated over recursively
just like the top-level form object.
When a form is submitted in the "old" format (as the query string or as a single
data part of type :mimetype:`application/x-www-form-urlencoded`), the items will
actually be instances of the class :class:`MiniFieldStorage`. In this case, the
:attr:`!list`, :attr:`!file`, and :attr:`filename` attributes are always ``None``.
A form submitted via POST that also has a query string will contain both
:class:`FieldStorage` and :class:`MiniFieldStorage` items.
.. versionchanged:: 3.4
The :attr:`~FieldStorage.file` attribute is automatically closed upon the
garbage collection of the creating :class:`FieldStorage` instance.
.. versionchanged:: 3.5
Added support for the context management protocol to the
:class:`FieldStorage` class.
Higher Level Interface
----------------------
The previous section explains how to read CGI form data using the
:class:`FieldStorage` class. This section describes a higher level interface
which was added to this class to allow one to do it in a more readable and
intuitive way. The interface doesn't make the techniques described in previous
sections obsolete --- they are still useful to process file uploads efficiently,
for example.
.. XXX: Is this true ?
The interface consists of two simple methods. Using the methods you can process
form data in a generic way, without the need to worry whether only one or more
values were posted under one name.
In the previous section, you learned to write following code anytime you
expected a user to post more than one value under one name::
item = form.getvalue("item")
if isinstance(item, list):
# The user is requesting more than one item.
else:
# The user is requesting only one item.
This situation is common for example when a form contains a group of multiple
checkboxes with the same name::
<input type="checkbox" name="item" value="1" />
<input type="checkbox" name="item" value="2" />
In most situations, however, there's only one form control with a particular
name in a form and then you expect and need only one value associated with this
name. So you write a script containing for example this code::
user = form.getvalue("user").upper()
The problem with the code is that you should never expect that a client will
provide valid input to your scripts. For example, if a curious user appends
another ``user=foo`` pair to the query string, then the script would crash,
because in this situation the ``getvalue("user")`` method call returns a list
instead of a string. Calling the :meth:`~str.upper` method on a list is not valid
(since lists do not have a method of this name) and results in an
:exc:`AttributeError` exception.
Therefore, the appropriate way to read form data values was to always use the
code which checks whether the obtained value is a single value or a list of
values. That's annoying and leads to less readable scripts.
A more convenient approach is to use the methods :meth:`~FieldStorage.getfirst`
and :meth:`~FieldStorage.getlist` provided by this higher level interface.
.. method:: FieldStorage.getfirst(name, default=None)
This method always returns only one value associated with form field *name*.
The method returns only the first value in case that more values were posted
under such name. Please note that the order in which the values are received
may vary from browser to browser and should not be counted on. [#]_ If no such
form field or value exists then the method returns the value specified by the
optional parameter *default*. This parameter defaults to ``None`` if not
specified.
.. method:: FieldStorage.getlist(name)
This method always returns a list of values associated with form field *name*.
The method returns an empty list if no such form field or value exists for
*name*. It returns a list consisting of one item if only one such value exists.
Using these methods you can write nice compact code::
import cgi
form = cgi.FieldStorage()
user = form.getfirst("user", "").upper() # This way it's safe.
for item in form.getlist("item"):
do_something(item)
.. _functions-in-cgi-module:
Functions
---------
These are useful if you want more control, or if you want to employ some of the
algorithms implemented in this module in other circumstances.
.. function:: parse(fp=None, environ=os.environ, keep_blank_values=False, strict_parsing=False, separator="&")
Parse a query in the environment or from a file (the file defaults to
``sys.stdin``). The *keep_blank_values*, *strict_parsing* and *separator* parameters are
passed to :func:`urllib.parse.parse_qs` unchanged.
.. function:: parse_qs(qs, keep_blank_values=False, strict_parsing=False)
This function is deprecated in this module. Use :func:`urllib.parse.parse_qs`
instead. It is maintained here only for backward compatibility.
.. function:: parse_qsl(qs, keep_blank_values=False, strict_parsing=False)
This function is deprecated in this module. Use :func:`urllib.parse.parse_qsl`
instead. It is maintained here only for backward compatibility.
.. function:: parse_multipart(fp, pdict)
Parse input of type :mimetype:`multipart/form-data` (for file uploads).
Arguments are *fp* for the input file and *pdict* for a dictionary containing
other parameters in the :mailheader:`Content-Type` header.
Returns a dictionary just like :func:`urllib.parse.parse_qs` keys are the field names, each
value is a list of values for that field. This is easy to use but not much good
if you are expecting megabytes to be uploaded --- in that case, use the
:class:`FieldStorage` class instead which is much more flexible.
Note that this does not parse nested multipart parts --- use
:class:`FieldStorage` for that.
.. versionchanged:: 3.6.13
Added the *separator* parameter.
.. function:: parse_header(string)
Parse a MIME header (such as :mailheader:`Content-Type`) into a main value and a
dictionary of parameters.
.. function:: test()
Robust test CGI script, usable as main program. Writes minimal HTTP headers and
formats all information provided to the script in HTML form.
.. function:: print_environ()
Format the shell environment in HTML.
.. function:: print_form(form)
Format a form in HTML.
.. function:: print_directory()
Format the current directory in HTML.
.. function:: print_environ_usage()
Print a list of useful (used by CGI) environment variables in HTML.
.. function:: escape(s, quote=False)
Convert the characters ``'&'``, ``'<'`` and ``'>'`` in string *s* to HTML-safe
sequences. Use this if you need to display text that might contain such
characters in HTML. If the optional flag *quote* is true, the quotation mark
character (``"``) is also translated; this helps for inclusion in an HTML
attribute value delimited by double quotes, as in ``<a href="...">``. Note
that single quotes are never translated.
.. deprecated:: 3.2
This function is unsafe because *quote* is false by default, and therefore
deprecated. Use :func:`html.escape` instead.
.. _cgi-security:
Caring about security
---------------------
.. index:: pair: CGI; security
There's one important rule: if you invoke an external program (via the
:func:`os.system` or :func:`os.popen` functions. or others with similar
functionality), make very sure you don't pass arbitrary strings received from
the client to the shell. This is a well-known security hole whereby clever
hackers anywhere on the Web can exploit a gullible CGI script to invoke
arbitrary shell commands. Even parts of the URL or field names cannot be
trusted, since the request doesn't have to come from your form!
To be on the safe side, if you must pass a string gotten from a form to a shell
command, you should make sure the string contains only alphanumeric characters,
dashes, underscores, and periods.
Installing your CGI script on a Unix system
-------------------------------------------
Read the documentation for your HTTP server and check with your local system
administrator to find the directory where CGI scripts should be installed;
usually this is in a directory :file:`cgi-bin` in the server tree.
Make sure that your script is readable and executable by "others"; the Unix file
mode should be ``0o755`` octal (use ``chmod 0755 filename``). Make sure that the
first line of the script contains ``#!`` starting in column 1 followed by the
pathname of the Python interpreter, for instance::
#!/usr/local/bin/python
Make sure the Python interpreter exists and is executable by "others".
Make sure that any files your script needs to read or write are readable or
writable, respectively, by "others" --- their mode should be ``0o644`` for
readable and ``0o666`` for writable. This is because, for security reasons, the
HTTP server executes your script as user "nobody", without any special
privileges. It can only read (write, execute) files that everybody can read
(write, execute). The current directory at execution time is also different (it
is usually the server's cgi-bin directory) and the set of environment variables
is also different from what you get when you log in. In particular, don't count
on the shell's search path for executables (:envvar:`PATH`) or the Python module
search path (:envvar:`PYTHONPATH`) to be set to anything interesting.
If you need to load modules from a directory which is not on Python's default
module search path, you can change the path in your script, before importing
other modules. For example::
import sys
sys.path.insert(0, "/usr/home/joe/lib/python")
sys.path.insert(0, "/usr/local/lib/python")
(This way, the directory inserted last will be searched first!)
Instructions for non-Unix systems will vary; check your HTTP server's
documentation (it will usually have a section on CGI scripts).
Testing your CGI script
-----------------------
Unfortunately, a CGI script will generally not run when you try it from the
command line, and a script that works perfectly from the command line may fail
mysteriously when run from the server. There's one reason why you should still
test your script from the command line: if it contains a syntax error, the
Python interpreter won't execute it at all, and the HTTP server will most likely
send a cryptic error to the client.
Assuming your script has no syntax errors, yet it does not work, you have no
choice but to read the next section.
Debugging CGI scripts
---------------------
.. index:: pair: CGI; debugging
First of all, check for trivial installation errors --- reading the section
above on installing your CGI script carefully can save you a lot of time. If
you wonder whether you have understood the installation procedure correctly, try
installing a copy of this module file (:file:`cgi.py`) as a CGI script. When
invoked as a script, the file will dump its environment and the contents of the
form in HTML form. Give it the right mode etc, and send it a request. If it's
installed in the standard :file:`cgi-bin` directory, it should be possible to
send it a request by entering a URL into your browser of the form:
.. code-block:: none
http://yourhostname/cgi-bin/cgi.py?name=Joe+Blow&addr=At+Home
If this gives an error of type 404, the server cannot find the script -- perhaps
you need to install it in a different directory. If it gives another error,
there's an installation problem that you should fix before trying to go any
further. If you get a nicely formatted listing of the environment and form
content (in this example, the fields should be listed as "addr" with value "At
Home" and "name" with value "Joe Blow"), the :file:`cgi.py` script has been
installed correctly. If you follow the same procedure for your own script, you
should now be able to debug it.
The next step could be to call the :mod:`cgi` module's :func:`test` function
from your script: replace its main code with the single statement ::
cgi.test()
This should produce the same results as those gotten from installing the
:file:`cgi.py` file itself.
When an ordinary Python script raises an unhandled exception (for whatever
reason: of a typo in a module name, a file that can't be opened, etc.), the
Python interpreter prints a nice traceback and exits. While the Python
interpreter will still do this when your CGI script raises an exception, most
likely the traceback will end up in one of the HTTP server's log files, or be
discarded altogether.
Fortunately, once you have managed to get your script to execute *some* code,
you can easily send tracebacks to the Web browser using the :mod:`cgitb` module.
If you haven't done so already, just add the lines::
import cgitb
cgitb.enable()
to the top of your script. Then try running it again; when a problem occurs,
you should see a detailed report that will likely make apparent the cause of the
crash.
If you suspect that there may be a problem in importing the :mod:`cgitb` module,
you can use an even more robust approach (which only uses built-in modules)::
import sys
sys.stderr = sys.stdout
print("Content-Type: text/plain")
print()
...your code here...
This relies on the Python interpreter to print the traceback. The content type
of the output is set to plain text, which disables all HTML processing. If your
script works, the raw HTML will be displayed by your client. If it raises an
exception, most likely after the first two lines have been printed, a traceback
will be displayed. Because no HTML interpretation is going on, the traceback
will be readable.
Common problems and solutions
-----------------------------
* Most HTTP servers buffer the output from CGI scripts until the script is
completed. This means that it is not possible to display a progress report on
the client's display while the script is running.
* Check the installation instructions above.
* Check the HTTP server's log files. (``tail -f logfile`` in a separate window
may be useful!)
* Always check a script for syntax errors first, by doing something like
``python script.py``.
* If your script does not have any syntax errors, try adding ``import cgitb;
cgitb.enable()`` to the top of the script.
* When invoking external programs, make sure they can be found. Usually, this
means using absolute path names --- :envvar:`PATH` is usually not set to a very
useful value in a CGI script.
* When reading or writing external files, make sure they can be read or written
by the userid under which your CGI script will be running: this is typically the
userid under which the web server is running, or some explicitly specified
userid for a web server's ``suexec`` feature.
* Don't try to give a CGI script a set-uid mode. This doesn't work on most
systems, and is a security liability as well.
.. rubric:: Footnotes
.. [#] Note that some recent versions of the HTML specification do state what
order the field values should be supplied in, but knowing whether a request
was received from a conforming browser, or even from a browser at all, is
tedious and error-prone.

66
third_party/python/Doc/library/cgitb.rst vendored Normal file
View file

@ -0,0 +1,66 @@
:mod:`cgitb` --- Traceback manager for CGI scripts
==================================================
.. module:: cgitb
:synopsis: Configurable traceback handler for CGI scripts.
.. moduleauthor:: Ka-Ping Yee <ping@lfw.org>
.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
**Source code:** :source:`Lib/cgitb.py`
.. index::
single: CGI; exceptions
single: CGI; tracebacks
single: exceptions; in CGI scripts
single: tracebacks; in CGI scripts
--------------
The :mod:`cgitb` module provides a special exception handler for Python scripts.
(Its name is a bit misleading. It was originally designed to display extensive
traceback information in HTML for CGI scripts. It was later generalized to also
display this information in plain text.) After this module is activated, if an
uncaught exception occurs, a detailed, formatted report will be displayed. The
report includes a traceback showing excerpts of the source code for each level,
as well as the values of the arguments and local variables to currently running
functions, to help you debug the problem. Optionally, you can save this
information to a file instead of sending it to the browser.
To enable this feature, simply add this to the top of your CGI script::
import cgitb
cgitb.enable()
The options to the :func:`enable` function control whether the report is
displayed in the browser and whether the report is logged to a file for later
analysis.
.. function:: enable(display=1, logdir=None, context=5, format="html")
.. index:: single: excepthook() (in module sys)
This function causes the :mod:`cgitb` module to take over the interpreter's
default handling for exceptions by setting the value of :attr:`sys.excepthook`.
The optional argument *display* defaults to ``1`` and can be set to ``0`` to
suppress sending the traceback to the browser. If the argument *logdir* is
present, the traceback reports are written to files. The value of *logdir*
should be a directory where these files will be placed. The optional argument
*context* is the number of lines of context to display around the current line
of source code in the traceback; this defaults to ``5``. If the optional
argument *format* is ``"html"``, the output is formatted as HTML. Any other
value forces plain text output. The default value is ``"html"``.
.. function:: handler(info=None)
This function handles an exception using the default settings (that is, show a
report in the browser, but don't log to a file). This can be used when you've
caught an exception and want to report it using :mod:`cgitb`. The optional
*info* argument should be a 3-tuple containing an exception type, exception
value, and traceback object, exactly like the tuple returned by
:func:`sys.exc_info`. If the *info* argument is not supplied, the current
exception is obtained from :func:`sys.exc_info`.

137
third_party/python/Doc/library/chunk.rst vendored Normal file
View file

@ -0,0 +1,137 @@
:mod:`chunk` --- Read IFF chunked data
======================================
.. module:: chunk
:synopsis: Module to read IFF chunks.
.. moduleauthor:: Sjoerd Mullender <sjoerd@acm.org>
.. sectionauthor:: Sjoerd Mullender <sjoerd@acm.org>
**Source code:** :source:`Lib/chunk.py`
.. index::
single: Audio Interchange File Format
single: AIFF
single: AIFF-C
single: Real Media File Format
single: RMFF
--------------
This module provides an interface for reading files that use EA IFF 85 chunks.
[#]_ This format is used in at least the Audio Interchange File Format
(AIFF/AIFF-C) and the Real Media File Format (RMFF). The WAVE audio file format
is closely related and can also be read using this module.
A chunk has the following structure:
+---------+--------+-------------------------------+
| Offset | Length | Contents |
+=========+========+===============================+
| 0 | 4 | Chunk ID |
+---------+--------+-------------------------------+
| 4 | 4 | Size of chunk in big-endian |
| | | byte order, not including the |
| | | header |
+---------+--------+-------------------------------+
| 8 | *n* | Data bytes, where *n* is the |
| | | size given in the preceding |
| | | field |
+---------+--------+-------------------------------+
| 8 + *n* | 0 or 1 | Pad byte needed if *n* is odd |
| | | and chunk alignment is used |
+---------+--------+-------------------------------+
The ID is a 4-byte string which identifies the type of chunk.
The size field (a 32-bit value, encoded using big-endian byte order) gives the
size of the chunk data, not including the 8-byte header.
Usually an IFF-type file consists of one or more chunks. The proposed usage of
the :class:`Chunk` class defined here is to instantiate an instance at the start
of each chunk and read from the instance until it reaches the end, after which a
new instance can be instantiated. At the end of the file, creating a new
instance will fail with an :exc:`EOFError` exception.
.. class:: Chunk(file, align=True, bigendian=True, inclheader=False)
Class which represents a chunk. The *file* argument is expected to be a
file-like object. An instance of this class is specifically allowed. The
only method that is needed is :meth:`~io.IOBase.read`. If the methods
:meth:`~io.IOBase.seek` and :meth:`~io.IOBase.tell` are present and don't
raise an exception, they are also used.
If these methods are present and raise an exception, they are expected to not
have altered the object. If the optional argument *align* is true, chunks
are assumed to be aligned on 2-byte boundaries. If *align* is false, no
alignment is assumed. The default value is true. If the optional argument
*bigendian* is false, the chunk size is assumed to be in little-endian order.
This is needed for WAVE audio files. The default value is true. If the
optional argument *inclheader* is true, the size given in the chunk header
includes the size of the header. The default value is false.
A :class:`Chunk` object supports the following methods:
.. method:: getname()
Returns the name (ID) of the chunk. This is the first 4 bytes of the
chunk.
.. method:: getsize()
Returns the size of the chunk.
.. method:: close()
Close and skip to the end of the chunk. This does not close the
underlying file.
The remaining methods will raise :exc:`OSError` if called after the
:meth:`close` method has been called. Before Python 3.3, they used to
raise :exc:`IOError`, now an alias of :exc:`OSError`.
.. method:: isatty()
Returns ``False``.
.. method:: seek(pos, whence=0)
Set the chunk's current position. The *whence* argument is optional and
defaults to ``0`` (absolute file positioning); other values are ``1``
(seek relative to the current position) and ``2`` (seek relative to the
file's end). There is no return value. If the underlying file does not
allow seek, only forward seeks are allowed.
.. method:: tell()
Return the current position into the chunk.
.. method:: read(size=-1)
Read at most *size* bytes from the chunk (less if the read hits the end of
the chunk before obtaining *size* bytes). If the *size* argument is
negative or omitted, read all data until the end of the chunk. An empty
bytes object is returned when the end of the chunk is encountered
immediately.
.. method:: skip()
Skip to the end of the chunk. All further calls to :meth:`read` for the
chunk will return ``b''``. If you are not interested in the contents of
the chunk, this method should be called so that the file points to the
start of the next chunk.
.. rubric:: Footnotes
.. [#] "EA IFF 85" Standard for Interchange Format Files, Jerry Morrison, Electronic
Arts, January 1985.

312
third_party/python/Doc/library/cmath.rst vendored Normal file
View file

@ -0,0 +1,312 @@
:mod:`cmath` --- Mathematical functions for complex numbers
===========================================================
.. module:: cmath
:synopsis: Mathematical functions for complex numbers.
--------------
This module is always available. It provides access to mathematical functions
for complex numbers. The functions in this module accept integers,
floating-point numbers or complex numbers as arguments. They will also accept
any Python object that has either a :meth:`__complex__` or a :meth:`__float__`
method: these methods are used to convert the object to a complex or
floating-point number, respectively, and the function is then applied to the
result of the conversion.
.. note::
On platforms with hardware and system-level support for signed
zeros, functions involving branch cuts are continuous on *both*
sides of the branch cut: the sign of the zero distinguishes one
side of the branch cut from the other. On platforms that do not
support signed zeros the continuity is as specified below.
Conversions to and from polar coordinates
-----------------------------------------
A Python complex number ``z`` is stored internally using *rectangular*
or *Cartesian* coordinates. It is completely determined by its *real
part* ``z.real`` and its *imaginary part* ``z.imag``. In other
words::
z == z.real + z.imag*1j
*Polar coordinates* give an alternative way to represent a complex
number. In polar coordinates, a complex number *z* is defined by the
modulus *r* and the phase angle *phi*. The modulus *r* is the distance
from *z* to the origin, while the phase *phi* is the counterclockwise
angle, measured in radians, from the positive x-axis to the line
segment that joins the origin to *z*.
The following functions can be used to convert from the native
rectangular coordinates to polar coordinates and back.
.. function:: phase(x)
Return the phase of *x* (also known as the *argument* of *x*), as a
float. ``phase(x)`` is equivalent to ``math.atan2(x.imag,
x.real)``. The result lies in the range [-π, π], and the branch
cut for this operation lies along the negative real axis,
continuous from above. On systems with support for signed zeros
(which includes most systems in current use), this means that the
sign of the result is the same as the sign of ``x.imag``, even when
``x.imag`` is zero::
>>> phase(complex(-1.0, 0.0))
3.141592653589793
>>> phase(complex(-1.0, -0.0))
-3.141592653589793
.. note::
The modulus (absolute value) of a complex number *x* can be
computed using the built-in :func:`abs` function. There is no
separate :mod:`cmath` module function for this operation.
.. function:: polar(x)
Return the representation of *x* in polar coordinates. Returns a
pair ``(r, phi)`` where *r* is the modulus of *x* and phi is the
phase of *x*. ``polar(x)`` is equivalent to ``(abs(x),
phase(x))``.
.. function:: rect(r, phi)
Return the complex number *x* with polar coordinates *r* and *phi*.
Equivalent to ``r * (math.cos(phi) + math.sin(phi)*1j)``.
Power and logarithmic functions
-------------------------------
.. function:: exp(x)
Return the exponential value ``e**x``.
.. function:: log(x[, base])
Returns the logarithm of *x* to the given *base*. If the *base* is not
specified, returns the natural logarithm of *x*. There is one branch cut, from 0
along the negative real axis to -∞, continuous from above.
.. function:: log10(x)
Return the base-10 logarithm of *x*. This has the same branch cut as
:func:`log`.
.. function:: sqrt(x)
Return the square root of *x*. This has the same branch cut as :func:`log`.
Trigonometric functions
-----------------------
.. function:: acos(x)
Return the arc cosine of *x*. There are two branch cuts: One extends right from
1 along the real axis to ∞, continuous from below. The other extends left from
-1 along the real axis to -∞, continuous from above.
.. function:: asin(x)
Return the arc sine of *x*. This has the same branch cuts as :func:`acos`.
.. function:: atan(x)
Return the arc tangent of *x*. There are two branch cuts: One extends from
``1j`` along the imaginary axis to ``∞j``, continuous from the right. The
other extends from ``-1j`` along the imaginary axis to ``-∞j``, continuous
from the left.
.. function:: cos(x)
Return the cosine of *x*.
.. function:: sin(x)
Return the sine of *x*.
.. function:: tan(x)
Return the tangent of *x*.
Hyperbolic functions
--------------------
.. function:: acosh(x)
Return the inverse hyperbolic cosine of *x*. There is one branch cut,
extending left from 1 along the real axis to -∞, continuous from above.
.. function:: asinh(x)
Return the inverse hyperbolic sine of *x*. There are two branch cuts:
One extends from ``1j`` along the imaginary axis to ``∞j``,
continuous from the right. The other extends from ``-1j`` along
the imaginary axis to ``-∞j``, continuous from the left.
.. function:: atanh(x)
Return the inverse hyperbolic tangent of *x*. There are two branch cuts: One
extends from ``1`` along the real axis to ``∞``, continuous from below. The
other extends from ``-1`` along the real axis to ``-∞``, continuous from
above.
.. function:: cosh(x)
Return the hyperbolic cosine of *x*.
.. function:: sinh(x)
Return the hyperbolic sine of *x*.
.. function:: tanh(x)
Return the hyperbolic tangent of *x*.
Classification functions
------------------------
.. function:: isfinite(x)
Return ``True`` if both the real and imaginary parts of *x* are finite, and
``False`` otherwise.
.. versionadded:: 3.2
.. function:: isinf(x)
Return ``True`` if either the real or the imaginary part of *x* is an
infinity, and ``False`` otherwise.
.. function:: isnan(x)
Return ``True`` if either the real or the imaginary part of *x* is a NaN,
and ``False`` otherwise.
.. function:: isclose(a, b, *, rel_tol=1e-09, abs_tol=0.0)
Return ``True`` if the values *a* and *b* are close to each other and
``False`` otherwise.
Whether or not two values are considered close is determined according to
given absolute and relative tolerances.
*rel_tol* is the relative tolerance -- it is the maximum allowed difference
between *a* and *b*, relative to the larger absolute value of *a* or *b*.
For example, to set a tolerance of 5%, pass ``rel_tol=0.05``. The default
tolerance is ``1e-09``, which assures that the two values are the same
within about 9 decimal digits. *rel_tol* must be greater than zero.
*abs_tol* is the minimum absolute tolerance -- useful for comparisons near
zero. *abs_tol* must be at least zero.
If no errors occur, the result will be:
``abs(a-b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)``.
The IEEE 754 special values of ``NaN``, ``inf``, and ``-inf`` will be
handled according to IEEE rules. Specifically, ``NaN`` is not considered
close to any other value, including ``NaN``. ``inf`` and ``-inf`` are only
considered close to themselves.
.. versionadded:: 3.5
.. seealso::
:pep:`485` -- A function for testing approximate equality
Constants
---------
.. data:: pi
The mathematical constant *π*, as a float.
.. data:: e
The mathematical constant *e*, as a float.
.. data:: tau
The mathematical constant *τ*, as a float.
.. versionadded:: 3.6
.. data:: inf
Floating-point positive infinity. Equivalent to ``float('inf')``.
.. versionadded:: 3.6
.. data:: infj
Complex number with zero real part and positive infinity imaginary
part. Equivalent to ``complex(0.0, float('inf'))``.
.. versionadded:: 3.6
.. data:: nan
A floating-point "not a number" (NaN) value. Equivalent to
``float('nan')``.
.. versionadded:: 3.6
.. data:: nanj
Complex number with zero real part and NaN imaginary part. Equivalent to
``complex(0.0, float('nan'))``.
.. versionadded:: 3.6
.. index:: module: math
Note that the selection of functions is similar, but not identical, to that in
module :mod:`math`. The reason for having two modules is that some users aren't
interested in complex numbers, and perhaps don't even know what they are. They
would rather have ``math.sqrt(-1)`` raise an exception than return a complex
number. Also note that the functions defined in :mod:`cmath` always return a
complex number, even if the answer can be expressed as a real number (in which
case the complex number has an imaginary part of zero).
A note on branch cuts: They are curves along which the given function fails to
be continuous. They are a necessary feature of many complex functions. It is
assumed that if you need to compute with complex functions, you will understand
about branch cuts. Consult almost any (not too elementary) book on complex
variables for enlightenment. For information of the proper choice of branch
cuts for numerical purposes, a good reference should be the following:
.. seealso::
Kahan, W: Branch cuts for complex elementary functions; or, Much ado about
nothing's sign bit. In Iserles, A., and Powell, M. (eds.), The state of the art
in numerical analysis. Clarendon Press (1987) pp165--211.

381
third_party/python/Doc/library/cmd.rst vendored Normal file
View file

@ -0,0 +1,381 @@
:mod:`cmd` --- Support for line-oriented command interpreters
=============================================================
.. module:: cmd
:synopsis: Build line-oriented command interpreters.
.. sectionauthor:: Eric S. Raymond <esr@snark.thyrsus.com>
**Source code:** :source:`Lib/cmd.py`
--------------
The :class:`Cmd` class provides a simple framework for writing line-oriented
command interpreters. These are often useful for test harnesses, administrative
tools, and prototypes that will later be wrapped in a more sophisticated
interface.
.. class:: Cmd(completekey='tab', stdin=None, stdout=None)
A :class:`Cmd` instance or subclass instance is a line-oriented interpreter
framework. There is no good reason to instantiate :class:`Cmd` itself; rather,
it's useful as a superclass of an interpreter class you define yourself in order
to inherit :class:`Cmd`'s methods and encapsulate action methods.
The optional argument *completekey* is the :mod:`readline` name of a completion
key; it defaults to :kbd:`Tab`. If *completekey* is not :const:`None` and
:mod:`readline` is available, command completion is done automatically.
The optional arguments *stdin* and *stdout* specify the input and output file
objects that the Cmd instance or subclass instance will use for input and
output. If not specified, they will default to :data:`sys.stdin` and
:data:`sys.stdout`.
If you want a given *stdin* to be used, make sure to set the instance's
:attr:`use_rawinput` attribute to ``False``, otherwise *stdin* will be
ignored.
.. _cmd-objects:
Cmd Objects
-----------
A :class:`Cmd` instance has the following methods:
.. method:: Cmd.cmdloop(intro=None)
Repeatedly issue a prompt, accept input, parse an initial prefix off the
received input, and dispatch to action methods, passing them the remainder of
the line as argument.
The optional argument is a banner or intro string to be issued before the first
prompt (this overrides the :attr:`intro` class attribute).
If the :mod:`readline` module is loaded, input will automatically inherit
:program:`bash`\ -like history-list editing (e.g. :kbd:`Control-P` scrolls back
to the last command, :kbd:`Control-N` forward to the next one, :kbd:`Control-F`
moves the cursor to the right non-destructively, :kbd:`Control-B` moves the
cursor to the left non-destructively, etc.).
An end-of-file on input is passed back as the string ``'EOF'``.
.. index::
single: ? (question mark); in a command interpreter
single: ! (exclamation); in a command interpreter
An interpreter instance will recognize a command name ``foo`` if and only if it
has a method :meth:`do_foo`. As a special case, a line beginning with the
character ``'?'`` is dispatched to the method :meth:`do_help`. As another
special case, a line beginning with the character ``'!'`` is dispatched to the
method :meth:`do_shell` (if such a method is defined).
This method will return when the :meth:`postcmd` method returns a true value.
The *stop* argument to :meth:`postcmd` is the return value from the command's
corresponding :meth:`do_\*` method.
If completion is enabled, completing commands will be done automatically, and
completing of commands args is done by calling :meth:`complete_foo` with
arguments *text*, *line*, *begidx*, and *endidx*. *text* is the string prefix
we are attempting to match: all returned matches must begin with it. *line* is
the current input line with leading whitespace removed, *begidx* and *endidx*
are the beginning and ending indexes of the prefix text, which could be used to
provide different completion depending upon which position the argument is in.
All subclasses of :class:`Cmd` inherit a predefined :meth:`do_help`. This
method, called with an argument ``'bar'``, invokes the corresponding method
:meth:`help_bar`, and if that is not present, prints the docstring of
:meth:`do_bar`, if available. With no argument, :meth:`do_help` lists all
available help topics (that is, all commands with corresponding
:meth:`help_\*` methods or commands that have docstrings), and also lists any
undocumented commands.
.. method:: Cmd.onecmd(str)
Interpret the argument as though it had been typed in response to the prompt.
This may be overridden, but should not normally need to be; see the
:meth:`precmd` and :meth:`postcmd` methods for useful execution hooks. The
return value is a flag indicating whether interpretation of commands by the
interpreter should stop. If there is a :meth:`do_\*` method for the command
*str*, the return value of that method is returned, otherwise the return value
from the :meth:`default` method is returned.
.. method:: Cmd.emptyline()
Method called when an empty line is entered in response to the prompt. If this
method is not overridden, it repeats the last nonempty command entered.
.. method:: Cmd.default(line)
Method called on an input line when the command prefix is not recognized. If
this method is not overridden, it prints an error message and returns.
.. method:: Cmd.completedefault(text, line, begidx, endidx)
Method called to complete an input line when no command-specific
:meth:`complete_\*` method is available. By default, it returns an empty list.
.. method:: Cmd.precmd(line)
Hook method executed just before the command line *line* is interpreted, but
after the input prompt is generated and issued. This method is a stub in
:class:`Cmd`; it exists to be overridden by subclasses. The return value is
used as the command which will be executed by the :meth:`onecmd` method; the
:meth:`precmd` implementation may re-write the command or simply return *line*
unchanged.
.. method:: Cmd.postcmd(stop, line)
Hook method executed just after a command dispatch is finished. This method is
a stub in :class:`Cmd`; it exists to be overridden by subclasses. *line* is the
command line which was executed, and *stop* is a flag which indicates whether
execution will be terminated after the call to :meth:`postcmd`; this will be the
return value of the :meth:`onecmd` method. The return value of this method will
be used as the new value for the internal flag which corresponds to *stop*;
returning false will cause interpretation to continue.
.. method:: Cmd.preloop()
Hook method executed once when :meth:`cmdloop` is called. This method is a stub
in :class:`Cmd`; it exists to be overridden by subclasses.
.. method:: Cmd.postloop()
Hook method executed once when :meth:`cmdloop` is about to return. This method
is a stub in :class:`Cmd`; it exists to be overridden by subclasses.
Instances of :class:`Cmd` subclasses have some public instance variables:
.. attribute:: Cmd.prompt
The prompt issued to solicit input.
.. attribute:: Cmd.identchars
The string of characters accepted for the command prefix.
.. attribute:: Cmd.lastcmd
The last nonempty command prefix seen.
.. attribute:: Cmd.cmdqueue
A list of queued input lines. The cmdqueue list is checked in
:meth:`cmdloop` when new input is needed; if it is nonempty, its elements
will be processed in order, as if entered at the prompt.
.. attribute:: Cmd.intro
A string to issue as an intro or banner. May be overridden by giving the
:meth:`cmdloop` method an argument.
.. attribute:: Cmd.doc_header
The header to issue if the help output has a section for documented commands.
.. attribute:: Cmd.misc_header
The header to issue if the help output has a section for miscellaneous help
topics (that is, there are :meth:`help_\*` methods without corresponding
:meth:`do_\*` methods).
.. attribute:: Cmd.undoc_header
The header to issue if the help output has a section for undocumented commands
(that is, there are :meth:`do_\*` methods without corresponding :meth:`help_\*`
methods).
.. attribute:: Cmd.ruler
The character used to draw separator lines under the help-message headers. If
empty, no ruler line is drawn. It defaults to ``'='``.
.. attribute:: Cmd.use_rawinput
A flag, defaulting to true. If true, :meth:`cmdloop` uses :func:`input` to
display a prompt and read the next command; if false, :meth:`sys.stdout.write`
and :meth:`sys.stdin.readline` are used. (This means that by importing
:mod:`readline`, on systems that support it, the interpreter will automatically
support :program:`Emacs`\ -like line editing and command-history keystrokes.)
.. _cmd-example:
Cmd Example
-----------
.. sectionauthor:: Raymond Hettinger <python at rcn dot com>
The :mod:`cmd` module is mainly useful for building custom shells that let a
user work with a program interactively.
This section presents a simple example of how to build a shell around a few of
the commands in the :mod:`turtle` module.
Basic turtle commands such as :meth:`~turtle.forward` are added to a
:class:`Cmd` subclass with method named :meth:`do_forward`. The argument is
converted to a number and dispatched to the turtle module. The docstring is
used in the help utility provided by the shell.
The example also includes a basic record and playback facility implemented with
the :meth:`~Cmd.precmd` method which is responsible for converting the input to
lowercase and writing the commands to a file. The :meth:`do_playback` method
reads the file and adds the recorded commands to the :attr:`cmdqueue` for
immediate playback::
import cmd, sys
from turtle import *
class TurtleShell(cmd.Cmd):
intro = 'Welcome to the turtle shell. Type help or ? to list commands.\n'
prompt = '(turtle) '
file = None
# ----- basic turtle commands -----
def do_forward(self, arg):
'Move the turtle forward by the specified distance: FORWARD 10'
forward(*parse(arg))
def do_right(self, arg):
'Turn turtle right by given number of degrees: RIGHT 20'
right(*parse(arg))
def do_left(self, arg):
'Turn turtle left by given number of degrees: LEFT 90'
left(*parse(arg))
def do_goto(self, arg):
'Move turtle to an absolute position with changing orientation. GOTO 100 200'
goto(*parse(arg))
def do_home(self, arg):
'Return turtle to the home position: HOME'
home()
def do_circle(self, arg):
'Draw circle with given radius an options extent and steps: CIRCLE 50'
circle(*parse(arg))
def do_position(self, arg):
'Print the current turtle position: POSITION'
print('Current position is %d %d\n' % position())
def do_heading(self, arg):
'Print the current turtle heading in degrees: HEADING'
print('Current heading is %d\n' % (heading(),))
def do_color(self, arg):
'Set the color: COLOR BLUE'
color(arg.lower())
def do_undo(self, arg):
'Undo (repeatedly) the last turtle action(s): UNDO'
def do_reset(self, arg):
'Clear the screen and return turtle to center: RESET'
reset()
def do_bye(self, arg):
'Stop recording, close the turtle window, and exit: BYE'
print('Thank you for using Turtle')
self.close()
bye()
return True
# ----- record and playback -----
def do_record(self, arg):
'Save future commands to filename: RECORD rose.cmd'
self.file = open(arg, 'w')
def do_playback(self, arg):
'Playback commands from a file: PLAYBACK rose.cmd'
self.close()
with open(arg) as f:
self.cmdqueue.extend(f.read().splitlines())
def precmd(self, line):
line = line.lower()
if self.file and 'playback' not in line:
print(line, file=self.file)
return line
def close(self):
if self.file:
self.file.close()
self.file = None
def parse(arg):
'Convert a series of zero or more numbers to an argument tuple'
return tuple(map(int, arg.split()))
if __name__ == '__main__':
TurtleShell().cmdloop()
Here is a sample session with the turtle shell showing the help functions, using
blank lines to repeat commands, and the simple record and playback facility:
.. code-block:: none
Welcome to the turtle shell. Type help or ? to list commands.
(turtle) ?
Documented commands (type help <topic>):
========================================
bye color goto home playback record right
circle forward heading left position reset undo
(turtle) help forward
Move the turtle forward by the specified distance: FORWARD 10
(turtle) record spiral.cmd
(turtle) position
Current position is 0 0
(turtle) heading
Current heading is 0
(turtle) reset
(turtle) circle 20
(turtle) right 30
(turtle) circle 40
(turtle) right 30
(turtle) circle 60
(turtle) right 30
(turtle) circle 80
(turtle) right 30
(turtle) circle 100
(turtle) right 30
(turtle) circle 120
(turtle) right 30
(turtle) circle 120
(turtle) heading
Current heading is 180
(turtle) forward 100
(turtle)
(turtle) right 90
(turtle) forward 100
(turtle)
(turtle) right 90
(turtle) forward 400
(turtle) right 90
(turtle) forward 500
(turtle) right 90
(turtle) forward 400
(turtle) right 90
(turtle) forward 300
(turtle) playback spiral.cmd
Current position is 0 0
Current heading is 0
Current heading is 180
(turtle) bye
Thank you for using Turtle

184
third_party/python/Doc/library/code.rst vendored Normal file
View file

@ -0,0 +1,184 @@
:mod:`code` --- Interpreter base classes
========================================
.. module:: code
:synopsis: Facilities to implement read-eval-print loops.
**Source code:** :source:`Lib/code.py`
--------------
The ``code`` module provides facilities to implement read-eval-print loops in
Python. Two classes and convenience functions are included which can be used to
build applications which provide an interactive interpreter prompt.
.. class:: InteractiveInterpreter(locals=None)
This class deals with parsing and interpreter state (the user's namespace); it
does not deal with input buffering or prompting or input file naming (the
filename is always passed in explicitly). The optional *locals* argument
specifies the dictionary in which code will be executed; it defaults to a newly
created dictionary with key ``'__name__'`` set to ``'__console__'`` and key
``'__doc__'`` set to ``None``.
.. class:: InteractiveConsole(locals=None, filename="<console>")
Closely emulate the behavior of the interactive Python interpreter. This class
builds on :class:`InteractiveInterpreter` and adds prompting using the familiar
``sys.ps1`` and ``sys.ps2``, and input buffering.
.. function:: interact(banner=None, readfunc=None, local=None, exitmsg=None)
Convenience function to run a read-eval-print loop. This creates a new
instance of :class:`InteractiveConsole` and sets *readfunc* to be used as
the :meth:`InteractiveConsole.raw_input` method, if provided. If *local* is
provided, it is passed to the :class:`InteractiveConsole` constructor for
use as the default namespace for the interpreter loop. The :meth:`interact`
method of the instance is then run with *banner* and *exitmsg* passed as the
banner and exit message to use, if provided. The console object is discarded
after use.
.. versionchanged:: 3.6
Added *exitmsg* parameter.
.. function:: compile_command(source, filename="<input>", symbol="single")
This function is useful for programs that want to emulate Python's interpreter
main loop (a.k.a. the read-eval-print loop). The tricky part is to determine
when the user has entered an incomplete command that can be completed by
entering more text (as opposed to a complete command or a syntax error). This
function *almost* always makes the same decision as the real interpreter main
loop.
*source* is the source string; *filename* is the optional filename from which
source was read, defaulting to ``'<input>'``; and *symbol* is the optional
grammar start symbol, which should be either ``'single'`` (the default) or
``'eval'``.
Returns a code object (the same as ``compile(source, filename, symbol)``) if the
command is complete and valid; ``None`` if the command is incomplete; raises
:exc:`SyntaxError` if the command is complete and contains a syntax error, or
raises :exc:`OverflowError` or :exc:`ValueError` if the command contains an
invalid literal.
.. _interpreter-objects:
Interactive Interpreter Objects
-------------------------------
.. method:: InteractiveInterpreter.runsource(source, filename="<input>", symbol="single")
Compile and run some source in the interpreter. Arguments are the same as for
:func:`compile_command`; the default for *filename* is ``'<input>'``, and for
*symbol* is ``'single'``. One several things can happen:
* The input is incorrect; :func:`compile_command` raised an exception
(:exc:`SyntaxError` or :exc:`OverflowError`). A syntax traceback will be
printed by calling the :meth:`showsyntaxerror` method. :meth:`runsource`
returns ``False``.
* The input is incomplete, and more input is required; :func:`compile_command`
returned ``None``. :meth:`runsource` returns ``True``.
* The input is complete; :func:`compile_command` returned a code object. The
code is executed by calling the :meth:`runcode` (which also handles run-time
exceptions, except for :exc:`SystemExit`). :meth:`runsource` returns ``False``.
The return value can be used to decide whether to use ``sys.ps1`` or ``sys.ps2``
to prompt the next line.
.. method:: InteractiveInterpreter.runcode(code)
Execute a code object. When an exception occurs, :meth:`showtraceback` is called
to display a traceback. All exceptions are caught except :exc:`SystemExit`,
which is allowed to propagate.
A note about :exc:`KeyboardInterrupt`: this exception may occur elsewhere in
this code, and may not always be caught. The caller should be prepared to deal
with it.
.. method:: InteractiveInterpreter.showsyntaxerror(filename=None)
Display the syntax error that just occurred. This does not display a stack
trace because there isn't one for syntax errors. If *filename* is given, it is
stuffed into the exception instead of the default filename provided by Python's
parser, because it always uses ``'<string>'`` when reading from a string. The
output is written by the :meth:`write` method.
.. method:: InteractiveInterpreter.showtraceback()
Display the exception that just occurred. We remove the first stack item
because it is within the interpreter object implementation. The output is
written by the :meth:`write` method.
.. versionchanged:: 3.5 The full chained traceback is displayed instead
of just the primary traceback.
.. method:: InteractiveInterpreter.write(data)
Write a string to the standard error stream (``sys.stderr``). Derived classes
should override this to provide the appropriate output handling as needed.
.. _console-objects:
Interactive Console Objects
---------------------------
The :class:`InteractiveConsole` class is a subclass of
:class:`InteractiveInterpreter`, and so offers all the methods of the
interpreter objects as well as the following additions.
.. method:: InteractiveConsole.interact(banner=None, exitmsg=None)
Closely emulate the interactive Python console. The optional *banner* argument
specify the banner to print before the first interaction; by default it prints a
banner similar to the one printed by the standard Python interpreter, followed
by the class name of the console object in parentheses (so as not to confuse
this with the real interpreter -- since it's so close!).
The optional *exitmsg* argument specifies an exit message printed when exiting.
Pass the empty string to suppress the exit message. If *exitmsg* is not given or
``None``, a default message is printed.
.. versionchanged:: 3.4
To suppress printing any banner, pass an empty string.
.. versionchanged:: 3.6
Print an exit message when exiting.
.. method:: InteractiveConsole.push(line)
Push a line of source text to the interpreter. The line should not have a
trailing newline; it may have internal newlines. The line is appended to a
buffer and the interpreter's :meth:`runsource` method is called with the
concatenated contents of the buffer as source. If this indicates that the
command was executed or invalid, the buffer is reset; otherwise, the command is
incomplete, and the buffer is left as it was after the line was appended. The
return value is ``True`` if more input is required, ``False`` if the line was
dealt with in some way (this is the same as :meth:`runsource`).
.. method:: InteractiveConsole.resetbuffer()
Remove any unhandled source text from the input buffer.
.. method:: InteractiveConsole.raw_input(prompt="")
Write a prompt and read a line. The returned line does not include the trailing
newline. When the user enters the EOF key sequence, :exc:`EOFError` is raised.
The base implementation reads from ``sys.stdin``; a subclass may replace this
with a different implementation.

1502
third_party/python/Doc/library/codecs.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

72
third_party/python/Doc/library/codeop.rst vendored Normal file
View file

@ -0,0 +1,72 @@
:mod:`codeop` --- Compile Python code
=====================================
.. module:: codeop
:synopsis: Compile (possibly incomplete) Python code.
.. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il>
.. sectionauthor:: Michael Hudson <mwh@python.net>
**Source code:** :source:`Lib/codeop.py`
--------------
The :mod:`codeop` module provides utilities upon which the Python
read-eval-print loop can be emulated, as is done in the :mod:`code` module. As
a result, you probably don't want to use the module directly; if you want to
include such a loop in your program you probably want to use the :mod:`code`
module instead.
There are two parts to this job:
#. Being able to tell if a line of input completes a Python statement: in
short, telling whether to print '``>>>``' or '``...``' next.
#. Remembering which future statements the user has entered, so subsequent
input can be compiled with these in effect.
The :mod:`codeop` module provides a way of doing each of these things, and a way
of doing them both.
To do just the former:
.. function:: compile_command(source, filename="<input>", symbol="single")
Tries to compile *source*, which should be a string of Python code and return a
code object if *source* is valid Python code. In that case, the filename
attribute of the code object will be *filename*, which defaults to
``'<input>'``. Returns ``None`` if *source* is *not* valid Python code, but is a
prefix of valid Python code.
If there is a problem with *source*, an exception will be raised.
:exc:`SyntaxError` is raised if there is invalid Python syntax, and
:exc:`OverflowError` or :exc:`ValueError` if there is an invalid literal.
The *symbol* argument determines whether *source* is compiled as a statement
(``'single'``, the default) or as an :term:`expression` (``'eval'``). Any
other value will cause :exc:`ValueError` to be raised.
.. note::
It is possible (but not likely) that the parser stops parsing with a
successful outcome before reaching the end of the source; in this case,
trailing symbols may be ignored instead of causing an error. For example,
a backslash followed by two newlines may be followed by arbitrary garbage.
This will be fixed once the API for the parser is better.
.. class:: Compile()
Instances of this class have :meth:`__call__` methods identical in signature to
the built-in function :func:`compile`, but with the difference that if the
instance compiles program text containing a :mod:`__future__` statement, the
instance 'remembers' and compiles all subsequent program texts with the
statement in force.
.. class:: CommandCompiler()
Instances of this class have :meth:`__call__` methods identical in signature to
:func:`compile_command`; the difference is that if the instance compiles program
text containing a ``__future__`` statement, the instance 'remembers' and
compiles all subsequent program texts with the statement in force.

305
third_party/python/Doc/library/collections.abc.rst vendored Normal file
View file

@ -0,0 +1,305 @@
:mod:`collections.abc` --- Abstract Base Classes for Containers
===============================================================
.. module:: collections.abc
:synopsis: Abstract base classes for containers
.. moduleauthor:: Raymond Hettinger <python at rcn.com>
.. sectionauthor:: Raymond Hettinger <python at rcn.com>
.. versionadded:: 3.3
Formerly, this module was part of the :mod:`collections` module.
**Source code:** :source:`Lib/_collections_abc.py`
.. testsetup:: *
from collections import *
import itertools
__name__ = '<doctest>'
--------------
This module provides :term:`abstract base classes <abstract base class>` that
can be used to test whether a class provides a particular interface; for
example, whether it is hashable or whether it is a mapping.
.. _collections-abstract-base-classes:
Collections Abstract Base Classes
---------------------------------
The collections module offers the following :term:`ABCs <abstract base class>`:
.. tabularcolumns:: |l|L|L|L|
========================== ====================== ======================= ====================================================
ABC Inherits from Abstract Methods Mixin Methods
========================== ====================== ======================= ====================================================
:class:`Container` ``__contains__``
:class:`Hashable` ``__hash__``
:class:`Iterable` ``__iter__``
:class:`Iterator` :class:`Iterable` ``__next__`` ``__iter__``
:class:`Reversible` :class:`Iterable` ``__reversed__``
:class:`Generator` :class:`Iterator` ``send``, ``throw`` ``close``, ``__iter__``, ``__next__``
:class:`Sized` ``__len__``
:class:`Callable` ``__call__``
:class:`Collection` :class:`Sized`, ``__contains__``,
:class:`Iterable`, ``__iter__``,
:class:`Container` ``__len__``
:class:`Sequence` :class:`Reversible`, ``__getitem__``, ``__contains__``, ``__iter__``, ``__reversed__``,
:class:`Collection` ``__len__`` ``index``, and ``count``
:class:`MutableSequence` :class:`Sequence` ``__getitem__``, Inherited :class:`Sequence` methods and
``__setitem__``, ``append``, ``reverse``, ``extend``, ``pop``,
``__delitem__``, ``remove``, and ``__iadd__``
``__len__``,
``insert``
:class:`ByteString` :class:`Sequence` ``__getitem__``, Inherited :class:`Sequence` methods
``__len__``
:class:`Set` :class:`Collection` ``__contains__``, ``__le__``, ``__lt__``, ``__eq__``, ``__ne__``,
``__iter__``, ``__gt__``, ``__ge__``, ``__and__``, ``__or__``,
``__len__`` ``__sub__``, ``__xor__``, and ``isdisjoint``
:class:`MutableSet` :class:`Set` ``__contains__``, Inherited :class:`Set` methods and
``__iter__``, ``clear``, ``pop``, ``remove``, ``__ior__``,
``__len__``, ``__iand__``, ``__ixor__``, and ``__isub__``
``add``,
``discard``
:class:`Mapping` :class:`Collection` ``__getitem__``, ``__contains__``, ``keys``, ``items``, ``values``,
``__iter__``, ``get``, ``__eq__``, and ``__ne__``
``__len__``
:class:`MutableMapping` :class:`Mapping` ``__getitem__``, Inherited :class:`Mapping` methods and
``__setitem__``, ``pop``, ``popitem``, ``clear``, ``update``,
``__delitem__``, and ``setdefault``
``__iter__``,
``__len__``
:class:`MappingView` :class:`Sized` ``__len__``
:class:`ItemsView` :class:`MappingView`, ``__contains__``,
:class:`Set` ``__iter__``
:class:`KeysView` :class:`MappingView`, ``__contains__``,
:class:`Set` ``__iter__``
:class:`ValuesView` :class:`MappingView` ``__contains__``, ``__iter__``
:class:`Awaitable` ``__await__``
:class:`Coroutine` :class:`Awaitable` ``send``, ``throw`` ``close``
:class:`AsyncIterable` ``__aiter__``
:class:`AsyncIterator` :class:`AsyncIterable` ``__anext__`` ``__aiter__``
:class:`AsyncGenerator` :class:`AsyncIterator` ``asend``, ``athrow`` ``aclose``, ``__aiter__``, ``__anext__``
========================== ====================== ======================= ====================================================
.. class:: Container
Hashable
Sized
Callable
ABCs for classes that provide respectively the methods :meth:`__contains__`,
:meth:`__hash__`, :meth:`__len__`, and :meth:`__call__`.
.. class:: Iterable
ABC for classes that provide the :meth:`__iter__` method.
Checking ``isinstance(obj, Iterable)`` detects classes that are registered
as :class:`Iterable` or that have an :meth:`__iter__` method, but it does
not detect classes that iterate with the :meth:`__getitem__` method.
The only reliable way to determine whether an object is :term:`iterable`
is to call ``iter(obj)``.
.. class:: Collection
ABC for sized iterable container classes.
.. versionadded:: 3.6
.. class:: Iterator
ABC for classes that provide the :meth:`~iterator.__iter__` and
:meth:`~iterator.__next__` methods. See also the definition of
:term:`iterator`.
.. class:: Reversible
ABC for iterable classes that also provide the :meth:`__reversed__`
method.
.. versionadded:: 3.6
.. class:: Generator
ABC for generator classes that implement the protocol defined in
:pep:`342` that extends iterators with the :meth:`~generator.send`,
:meth:`~generator.throw` and :meth:`~generator.close` methods.
See also the definition of :term:`generator`.
.. versionadded:: 3.5
.. class:: Sequence
MutableSequence
ByteString
ABCs for read-only and mutable :term:`sequences <sequence>`.
Implementation note: Some of the mixin methods, such as
:meth:`__iter__`, :meth:`__reversed__` and :meth:`index`, make
repeated calls to the underlying :meth:`__getitem__` method.
Consequently, if :meth:`__getitem__` is implemented with constant
access speed, the mixin methods will have linear performance;
however, if the underlying method is linear (as it would be with a
linked list), the mixins will have quadratic performance and will
likely need to be overridden.
.. versionchanged:: 3.5
The index() method added support for *stop* and *start*
arguments.
.. class:: Set
MutableSet
ABCs for read-only and mutable sets.
.. class:: Mapping
MutableMapping
ABCs for read-only and mutable :term:`mappings <mapping>`.
.. class:: MappingView
ItemsView
KeysView
ValuesView
ABCs for mapping, items, keys, and values :term:`views <dictionary view>`.
.. class:: Awaitable
ABC for :term:`awaitable` objects, which can be used in :keyword:`await`
expressions. Custom implementations must provide the :meth:`__await__`
method.
:term:`Coroutine` objects and instances of the
:class:`~collections.abc.Coroutine` ABC are all instances of this ABC.
.. note::
In CPython, generator-based coroutines (generators decorated with
:func:`types.coroutine` or :func:`asyncio.coroutine`) are
*awaitables*, even though they do not have an :meth:`__await__` method.
Using ``isinstance(gencoro, Awaitable)`` for them will return ``False``.
Use :func:`inspect.isawaitable` to detect them.
.. versionadded:: 3.5
.. class:: Coroutine
ABC for coroutine compatible classes. These implement the
following methods, defined in :ref:`coroutine-objects`:
:meth:`~coroutine.send`, :meth:`~coroutine.throw`, and
:meth:`~coroutine.close`. Custom implementations must also implement
:meth:`__await__`. All :class:`Coroutine` instances are also instances of
:class:`Awaitable`. See also the definition of :term:`coroutine`.
.. note::
In CPython, generator-based coroutines (generators decorated with
:func:`types.coroutine` or :func:`asyncio.coroutine`) are
*awaitables*, even though they do not have an :meth:`__await__` method.
Using ``isinstance(gencoro, Coroutine)`` for them will return ``False``.
Use :func:`inspect.isawaitable` to detect them.
.. versionadded:: 3.5
.. class:: AsyncIterable
ABC for classes that provide ``__aiter__`` method. See also the
definition of :term:`asynchronous iterable`.
.. versionadded:: 3.5
.. class:: AsyncIterator
ABC for classes that provide ``__aiter__`` and ``__anext__``
methods. See also the definition of :term:`asynchronous iterator`.
.. versionadded:: 3.5
.. class:: AsyncGenerator
ABC for asynchronous generator classes that implement the protocol
defined in :pep:`525` and :pep:`492`.
.. versionadded:: 3.6
These ABCs allow us to ask classes or instances if they provide
particular functionality, for example::
size = None
if isinstance(myvar, collections.abc.Sized):
size = len(myvar)
Several of the ABCs are also useful as mixins that make it easier to develop
classes supporting container APIs. For example, to write a class supporting
the full :class:`Set` API, it is only necessary to supply the three underlying
abstract methods: :meth:`__contains__`, :meth:`__iter__`, and :meth:`__len__`.
The ABC supplies the remaining methods such as :meth:`__and__` and
:meth:`isdisjoint`::
class ListBasedSet(collections.abc.Set):
''' Alternate set implementation favoring space over speed
and not requiring the set elements to be hashable. '''
def __init__(self, iterable):
self.elements = lst = []
for value in iterable:
if value not in lst:
lst.append(value)
def __iter__(self):
return iter(self.elements)
def __contains__(self, value):
return value in self.elements
def __len__(self):
return len(self.elements)
s1 = ListBasedSet('abcdef')
s2 = ListBasedSet('defghi')
overlap = s1 & s2 # The __and__() method is supported automatically
Notes on using :class:`Set` and :class:`MutableSet` as a mixin:
(1)
Since some set operations create new sets, the default mixin methods need
a way to create new instances from an iterable. The class constructor is
assumed to have a signature in the form ``ClassName(iterable)``.
That assumption is factored-out to an internal classmethod called
:meth:`_from_iterable` which calls ``cls(iterable)`` to produce a new set.
If the :class:`Set` mixin is being used in a class with a different
constructor signature, you will need to override :meth:`_from_iterable`
with a classmethod that can construct new instances from
an iterable argument.
(2)
To override the comparisons (presumably for speed, as the
semantics are fixed), redefine :meth:`__le__` and :meth:`__ge__`,
then the other operations will automatically follow suit.
(3)
The :class:`Set` mixin provides a :meth:`_hash` method to compute a hash value
for the set; however, :meth:`__hash__` is not defined because not all sets
are hashable or immutable. To add set hashability using mixins,
inherit from both :meth:`Set` and :meth:`Hashable`, then define
``__hash__ = Set._hash``.
.. seealso::
* `OrderedSet recipe <https://code.activestate.com/recipes/576694/>`_ for an
example built on :class:`MutableSet`.
* For more about ABCs, see the :mod:`abc` module and :pep:`3119`.

1184
third_party/python/Doc/library/collections.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

65
third_party/python/Doc/library/colorsys.rst vendored Normal file
View file

@ -0,0 +1,65 @@
:mod:`colorsys` --- Conversions between color systems
=====================================================
.. module:: colorsys
:synopsis: Conversion functions between RGB and other color systems.
.. sectionauthor:: David Ascher <da@python.net>
**Source code:** :source:`Lib/colorsys.py`
--------------
The :mod:`colorsys` module defines bidirectional conversions of color values
between colors expressed in the RGB (Red Green Blue) color space used in
computer monitors and three other coordinate systems: YIQ, HLS (Hue Lightness
Saturation) and HSV (Hue Saturation Value). Coordinates in all of these color
spaces are floating point values. In the YIQ space, the Y coordinate is between
0 and 1, but the I and Q coordinates can be positive or negative. In all other
spaces, the coordinates are all between 0 and 1.
.. seealso::
More information about color spaces can be found at
http://www.poynton.com/ColorFAQ.html and
https://www.cambridgeincolour.com/tutorials/color-spaces.htm.
The :mod:`colorsys` module defines the following functions:
.. function:: rgb_to_yiq(r, g, b)
Convert the color from RGB coordinates to YIQ coordinates.
.. function:: yiq_to_rgb(y, i, q)
Convert the color from YIQ coordinates to RGB coordinates.
.. function:: rgb_to_hls(r, g, b)
Convert the color from RGB coordinates to HLS coordinates.
.. function:: hls_to_rgb(h, l, s)
Convert the color from HLS coordinates to RGB coordinates.
.. function:: rgb_to_hsv(r, g, b)
Convert the color from RGB coordinates to HSV coordinates.
.. function:: hsv_to_rgb(h, s, v)
Convert the color from HSV coordinates to RGB coordinates.
Example::
>>> import colorsys
>>> colorsys.rgb_to_hsv(0.2, 0.4, 0.4)
(0.5, 0.5, 0.4)
>>> colorsys.hsv_to_rgb(0.5, 0.5, 0.4)
(0.2, 0.4, 0.4)

234
third_party/python/Doc/library/compileall.rst vendored Normal file
View file

@ -0,0 +1,234 @@
:mod:`compileall` --- Byte-compile Python libraries
===================================================
.. module:: compileall
:synopsis: Tools for byte-compiling all Python source files in a directory tree.
**Source code:** :source:`Lib/compileall.py`
--------------
This module provides some utility functions to support installing Python
libraries. These functions compile Python source files in a directory tree.
This module can be used to create the cached byte-code files at library
installation time, which makes them available for use even by users who don't
have write permission to the library directories.
Command-line use
----------------
This module can work as a script (using :program:`python -m compileall`) to
compile Python sources.
.. program:: compileall
.. cmdoption:: directory ...
file ...
Positional arguments are files to compile or directories that contain
source files, traversed recursively. If no argument is given, behave as if
the command line was ``-l <directories from sys.path>``.
.. cmdoption:: -l
Do not recurse into subdirectories, only compile source code files directly
contained in the named or implied directories.
.. cmdoption:: -f
Force rebuild even if timestamps are up-to-date.
.. cmdoption:: -q
Do not print the list of files compiled. If passed once, error messages will
still be printed. If passed twice (``-qq``), all output is suppressed.
.. cmdoption:: -d destdir
Directory prepended to the path to each file being compiled. This will
appear in compilation time tracebacks, and is also compiled in to the
byte-code file, where it will be used in tracebacks and other messages in
cases where the source file does not exist at the time the byte-code file is
executed.
.. cmdoption:: -x regex
regex is used to search the full path to each file considered for
compilation, and if the regex produces a match, the file is skipped.
.. cmdoption:: -i list
Read the file ``list`` and add each line that it contains to the list of
files and directories to compile. If ``list`` is ``-``, read lines from
``stdin``.
.. cmdoption:: -b
Write the byte-code files to their legacy locations and names, which may
overwrite byte-code files created by another version of Python. The default
is to write files to their :pep:`3147` locations and names, which allows
byte-code files from multiple versions of Python to coexist.
.. cmdoption:: -r
Control the maximum recursion level for subdirectories.
If this is given, then ``-l`` option will not be taken into account.
:program:`python -m compileall <directory> -r 0` is equivalent to
:program:`python -m compileall <directory> -l`.
.. cmdoption:: -j N
Use *N* workers to compile the files within the given directory.
If ``0`` is used, then the result of :func:`os.cpu_count()`
will be used.
.. versionchanged:: 3.2
Added the ``-i``, ``-b`` and ``-h`` options.
.. versionchanged:: 3.5
Added the ``-j``, ``-r``, and ``-qq`` options. ``-q`` option
was changed to a multilevel value. ``-b`` will always produce a
byte-code file ending in ``.pyc``, never ``.pyo``.
There is no command-line option to control the optimization level used by the
:func:`compile` function, because the Python interpreter itself already
provides the option: :program:`python -O -m compileall`.
Public functions
----------------
.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1)
Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
files along the way. Return a true value if all the files compiled successfully,
and a false value otherwise.
The *maxlevels* parameter is used to limit the depth of the recursion; it
defaults to ``10``.
If *ddir* is given, it is prepended to the path to each file being compiled
for use in compilation time tracebacks, and is also compiled in to the
byte-code file, where it will be used in tracebacks and other messages in
cases where the source file does not exist at the time the byte-code file is
executed.
If *force* is true, modules are re-compiled even if the timestamps are up to
date.
If *rx* is given, its search method is called on the complete path to each
file considered for compilation, and if it returns a true value, the file
is skipped.
If *quiet* is ``False`` or ``0`` (the default), the filenames and other
information are printed to standard out. Set to ``1``, only errors are
printed. Set to ``2``, all output is suppressed.
If *legacy* is true, byte-code files are written to their legacy locations
and names, which may overwrite byte-code files created by another version of
Python. The default is to write files to their :pep:`3147` locations and
names, which allows byte-code files from multiple versions of Python to
coexist.
*optimize* specifies the optimization level for the compiler. It is passed to
the built-in :func:`compile` function.
The argument *workers* specifies how many workers are used to
compile files in parallel. The default is to not use multiple workers.
If the platform can't use multiple workers and *workers* argument is given,
then sequential compilation will be used as a fallback. If *workers* is
lower than ``0``, a :exc:`ValueError` will be raised.
.. versionchanged:: 3.2
Added the *legacy* and *optimize* parameter.
.. versionchanged:: 3.5
Added the *workers* parameter.
.. versionchanged:: 3.5
*quiet* parameter was changed to a multilevel value.
.. versionchanged:: 3.5
The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
no matter what the value of *optimize* is.
.. versionchanged:: 3.6
Accepts a :term:`path-like object`.
.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1)
Compile the file with path *fullname*. Return a true value if the file
compiled successfully, and a false value otherwise.
If *ddir* is given, it is prepended to the path to the file being compiled
for use in compilation time tracebacks, and is also compiled in to the
byte-code file, where it will be used in tracebacks and other messages in
cases where the source file does not exist at the time the byte-code file is
executed.
If *rx* is given, its search method is passed the full path name to the
file being compiled, and if it returns a true value, the file is not
compiled and ``True`` is returned.
If *quiet* is ``False`` or ``0`` (the default), the filenames and other
information are printed to standard out. Set to ``1``, only errors are
printed. Set to ``2``, all output is suppressed.
If *legacy* is true, byte-code files are written to their legacy locations
and names, which may overwrite byte-code files created by another version of
Python. The default is to write files to their :pep:`3147` locations and
names, which allows byte-code files from multiple versions of Python to
coexist.
*optimize* specifies the optimization level for the compiler. It is passed to
the built-in :func:`compile` function.
.. versionadded:: 3.2
.. versionchanged:: 3.5
*quiet* parameter was changed to a multilevel value.
.. versionchanged:: 3.5
The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
no matter what the value of *optimize* is.
.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1)
Byte-compile all the :file:`.py` files found along ``sys.path``. Return a
true value if all the files compiled successfully, and a false value otherwise.
If *skip_curdir* is true (the default), the current directory is not included
in the search. All other parameters are passed to the :func:`compile_dir`
function. Note that unlike the other compile functions, ``maxlevels``
defaults to ``0``.
.. versionchanged:: 3.2
Added the *legacy* and *optimize* parameter.
.. versionchanged:: 3.5
*quiet* parameter was changed to a multilevel value.
.. versionchanged:: 3.5
The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
no matter what the value of *optimize* is.
To force a recompile of all the :file:`.py` files in the :file:`Lib/`
subdirectory and all its subdirectories::
import compileall
compileall.compile_dir('Lib/', force=True)
# Perform same compilation, excluding files in .svn directories.
import re
compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True)
# pathlib.Path objects can also be used.
import pathlib
compileall.compile_dir(pathlib.Path('Lib/'), force=True)
.. seealso::
Module :mod:`py_compile`
Byte-compile a single source file.

31
third_party/python/Doc/library/concurrency.rst vendored Normal file
View file

@ -0,0 +1,31 @@
.. _concurrency:
********************
Concurrent Execution
********************
The modules described in this chapter provide support for concurrent
execution of code. The appropriate choice of tool will depend on the
task to be executed (CPU bound vs IO bound) and preferred style of
development (event driven cooperative multitasking vs preemptive
multitasking). Here's an overview:
.. toctree::
threading.rst
multiprocessing.rst
concurrent.rst
concurrent.futures.rst
subprocess.rst
sched.rst
queue.rst
The following are support modules for some of the above services:
.. toctree::
dummy_threading.rst
_thread.rst
_dummy_thread.rst

445
third_party/python/Doc/library/concurrent.futures.rst vendored Normal file
View file

@ -0,0 +1,445 @@
:mod:`concurrent.futures` --- Launching parallel tasks
======================================================
.. module:: concurrent.futures
:synopsis: Execute computations concurrently using threads or processes.
.. versionadded:: 3.2
**Source code:** :source:`Lib/concurrent/futures/thread.py`
and :source:`Lib/concurrent/futures/process.py`
--------------
The :mod:`concurrent.futures` module provides a high-level interface for
asynchronously executing callables.
The asynchronous execution can be performed with threads, using
:class:`ThreadPoolExecutor`, or separate processes, using
:class:`ProcessPoolExecutor`. Both implement the same interface, which is
defined by the abstract :class:`Executor` class.
Executor Objects
----------------
.. class:: Executor
An abstract class that provides methods to execute calls asynchronously. It
should not be used directly, but through its concrete subclasses.
.. method:: submit(fn, *args, **kwargs)
Schedules the callable, *fn*, to be executed as ``fn(*args **kwargs)``
and returns a :class:`Future` object representing the execution of the
callable. ::
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(pow, 323, 1235)
print(future.result())
.. method:: map(func, *iterables, timeout=None, chunksize=1)
Similar to :func:`map(func, *iterables) <map>` except:
* the *iterables* are collected immediately rather than lazily;
* *func* is executed asynchronously and several calls to
*func* may be made concurrently.
The returned iterator raises a :exc:`concurrent.futures.TimeoutError`
if :meth:`~iterator.__next__` is called and the result isn't available
after *timeout* seconds from the original call to :meth:`Executor.map`.
*timeout* can be an int or a float. If *timeout* is not specified or
``None``, there is no limit to the wait time.
If a *func* call raises an exception, then that exception will be
raised when its value is retrieved from the iterator.
When using :class:`ProcessPoolExecutor`, this method chops *iterables*
into a number of chunks which it submits to the pool as separate
tasks. The (approximate) size of these chunks can be specified by
setting *chunksize* to a positive integer. For very long iterables,
using a large value for *chunksize* can significantly improve
performance compared to the default size of 1. With
:class:`ThreadPoolExecutor`, *chunksize* has no effect.
.. versionchanged:: 3.5
Added the *chunksize* argument.
.. method:: shutdown(wait=True)
Signal the executor that it should free any resources that it is using
when the currently pending futures are done executing. Calls to
:meth:`Executor.submit` and :meth:`Executor.map` made after shutdown will
raise :exc:`RuntimeError`.
If *wait* is ``True`` then this method will not return until all the
pending futures are done executing and the resources associated with the
executor have been freed. If *wait* is ``False`` then this method will
return immediately and the resources associated with the executor will be
freed when all pending futures are done executing. Regardless of the
value of *wait*, the entire Python program will not exit until all
pending futures are done executing.
You can avoid having to call this method explicitly if you use the
:keyword:`with` statement, which will shutdown the :class:`Executor`
(waiting as if :meth:`Executor.shutdown` were called with *wait* set to
``True``)::
import shutil
with ThreadPoolExecutor(max_workers=4) as e:
e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
e.submit(shutil.copy, 'src4.txt', 'dest4.txt')
ThreadPoolExecutor
------------------
:class:`ThreadPoolExecutor` is an :class:`Executor` subclass that uses a pool of
threads to execute calls asynchronously.
Deadlocks can occur when the callable associated with a :class:`Future` waits on
the results of another :class:`Future`. For example::
import time
def wait_on_b():
time.sleep(5)
print(b.result()) # b will never complete because it is waiting on a.
return 5
def wait_on_a():
time.sleep(5)
print(a.result()) # a will never complete because it is waiting on b.
return 6
executor = ThreadPoolExecutor(max_workers=2)
a = executor.submit(wait_on_b)
b = executor.submit(wait_on_a)
And::
def wait_on_future():
f = executor.submit(pow, 5, 2)
# This will never complete because there is only one worker thread and
# it is executing this function.
print(f.result())
executor = ThreadPoolExecutor(max_workers=1)
executor.submit(wait_on_future)
.. class:: ThreadPoolExecutor(max_workers=None, thread_name_prefix='')
An :class:`Executor` subclass that uses a pool of at most *max_workers*
threads to execute calls asynchronously.
.. versionchanged:: 3.5
If *max_workers* is ``None`` or
not given, it will default to the number of processors on the machine,
multiplied by ``5``, assuming that :class:`ThreadPoolExecutor` is often
used to overlap I/O instead of CPU work and the number of workers
should be higher than the number of workers
for :class:`ProcessPoolExecutor`.
.. versionadded:: 3.6
The *thread_name_prefix* argument was added to allow users to
control the :class:`threading.Thread` names for worker threads created by
the pool for easier debugging.
.. _threadpoolexecutor-example:
ThreadPoolExecutor Example
~~~~~~~~~~~~~~~~~~~~~~~~~~
::
import concurrent.futures
import urllib.request
URLS = ['http://www.foxnews.com/',
'http://www.cnn.com/',
'http://europe.wsj.com/',
'http://www.bbc.co.uk/',
'http://some-made-up-domain.com/']
# Retrieve a single page and report the URL and contents
def load_url(url, timeout):
with urllib.request.urlopen(url, timeout=timeout) as conn:
return conn.read()
# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
# Start the load operations and mark each future with its URL
future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
for future in concurrent.futures.as_completed(future_to_url):
url = future_to_url[future]
try:
data = future.result()
except Exception as exc:
print('%r generated an exception: %s' % (url, exc))
else:
print('%r page is %d bytes' % (url, len(data)))
ProcessPoolExecutor
-------------------
The :class:`ProcessPoolExecutor` class is an :class:`Executor` subclass that
uses a pool of processes to execute calls asynchronously.
:class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which
allows it to side-step the :term:`Global Interpreter Lock` but also means that
only picklable objects can be executed and returned.
The ``__main__`` module must be importable by worker subprocesses. This means
that :class:`ProcessPoolExecutor` will not work in the interactive interpreter.
Calling :class:`Executor` or :class:`Future` methods from a callable submitted
to a :class:`ProcessPoolExecutor` will result in deadlock.
.. class:: ProcessPoolExecutor(max_workers=None)
An :class:`Executor` subclass that executes calls asynchronously using a pool
of at most *max_workers* processes. If *max_workers* is ``None`` or not
given, it will default to the number of processors on the machine.
If *max_workers* is lower or equal to ``0``, then a :exc:`ValueError`
will be raised.
.. versionchanged:: 3.3
When one of the worker processes terminates abruptly, a
:exc:`BrokenProcessPool` error is now raised. Previously, behaviour
was undefined but operations on the executor or its futures would often
freeze or deadlock.
.. _processpoolexecutor-example:
ProcessPoolExecutor Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
import concurrent.futures
import math
PRIMES = [
112272535095293,
112582705942171,
112272535095293,
115280095190773,
115797848077099,
1099726899285419]
def is_prime(n):
if n % 2 == 0:
return False
sqrt_n = int(math.floor(math.sqrt(n)))
for i in range(3, sqrt_n + 1, 2):
if n % i == 0:
return False
return True
def main():
with concurrent.futures.ProcessPoolExecutor() as executor:
for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
print('%d is prime: %s' % (number, prime))
if __name__ == '__main__':
main()
Future Objects
--------------
The :class:`Future` class encapsulates the asynchronous execution of a callable.
:class:`Future` instances are created by :meth:`Executor.submit`.
.. class:: Future
Encapsulates the asynchronous execution of a callable. :class:`Future`
instances are created by :meth:`Executor.submit` and should not be created
directly except for testing.
.. method:: cancel()
Attempt to cancel the call. If the call is currently being executed and
cannot be cancelled then the method will return ``False``, otherwise the
call will be cancelled and the method will return ``True``.
.. method:: cancelled()
Return ``True`` if the call was successfully cancelled.
.. method:: running()
Return ``True`` if the call is currently being executed and cannot be
cancelled.
.. method:: done()
Return ``True`` if the call was successfully cancelled or finished
running.
.. method:: result(timeout=None)
Return the value returned by the call. If the call hasn't yet completed
then this method will wait up to *timeout* seconds. If the call hasn't
completed in *timeout* seconds, then a
:exc:`concurrent.futures.TimeoutError` will be raised. *timeout* can be
an int or float. If *timeout* is not specified or ``None``, there is no
limit to the wait time.
If the future is cancelled before completing then :exc:`.CancelledError`
will be raised.
If the call raised, this method will raise the same exception.
.. method:: exception(timeout=None)
Return the exception raised by the call. If the call hasn't yet
completed then this method will wait up to *timeout* seconds. If the
call hasn't completed in *timeout* seconds, then a
:exc:`concurrent.futures.TimeoutError` will be raised. *timeout* can be
an int or float. If *timeout* is not specified or ``None``, there is no
limit to the wait time.
If the future is cancelled before completing then :exc:`.CancelledError`
will be raised.
If the call completed without raising, ``None`` is returned.
.. method:: add_done_callback(fn)
Attaches the callable *fn* to the future. *fn* will be called, with the
future as its only argument, when the future is cancelled or finishes
running.
Added callables are called in the order that they were added and are
always called in a thread belonging to the process that added them. If
the callable raises an :exc:`Exception` subclass, it will be logged and
ignored. If the callable raises a :exc:`BaseException` subclass, the
behavior is undefined.
If the future has already completed or been cancelled, *fn* will be
called immediately.
The following :class:`Future` methods are meant for use in unit tests and
:class:`Executor` implementations.
.. method:: set_running_or_notify_cancel()
This method should only be called by :class:`Executor` implementations
before executing the work associated with the :class:`Future` and by unit
tests.
If the method returns ``False`` then the :class:`Future` was cancelled,
i.e. :meth:`Future.cancel` was called and returned `True`. Any threads
waiting on the :class:`Future` completing (i.e. through
:func:`as_completed` or :func:`wait`) will be woken up.
If the method returns ``True`` then the :class:`Future` was not cancelled
and has been put in the running state, i.e. calls to
:meth:`Future.running` will return `True`.
This method can only be called once and cannot be called after
:meth:`Future.set_result` or :meth:`Future.set_exception` have been
called.
.. method:: set_result(result)
Sets the result of the work associated with the :class:`Future` to
*result*.
This method should only be used by :class:`Executor` implementations and
unit tests.
.. method:: set_exception(exception)
Sets the result of the work associated with the :class:`Future` to the
:class:`Exception` *exception*.
This method should only be used by :class:`Executor` implementations and
unit tests.
Module Functions
----------------
.. function:: wait(fs, timeout=None, return_when=ALL_COMPLETED)
Wait for the :class:`Future` instances (possibly created by different
:class:`Executor` instances) given by *fs* to complete. Returns a named
2-tuple of sets. The first set, named ``done``, contains the futures that
completed (finished or were cancelled) before the wait completed. The second
set, named ``not_done``, contains uncompleted futures.
*timeout* can be used to control the maximum number of seconds to wait before
returning. *timeout* can be an int or float. If *timeout* is not specified
or ``None``, there is no limit to the wait time.
*return_when* indicates when this function should return. It must be one of
the following constants:
.. tabularcolumns:: |l|L|
+-----------------------------+----------------------------------------+
| Constant | Description |
+=============================+========================================+
| :const:`FIRST_COMPLETED` | The function will return when any |
| | future finishes or is cancelled. |
+-----------------------------+----------------------------------------+
| :const:`FIRST_EXCEPTION` | The function will return when any |
| | future finishes by raising an |
| | exception. If no future raises an |
| | exception then it is equivalent to |
| | :const:`ALL_COMPLETED`. |
+-----------------------------+----------------------------------------+
| :const:`ALL_COMPLETED` | The function will return when all |
| | futures finish or are cancelled. |
+-----------------------------+----------------------------------------+
.. function:: as_completed(fs, timeout=None)
Returns an iterator over the :class:`Future` instances (possibly created by
different :class:`Executor` instances) given by *fs* that yields futures as
they complete (finished or were cancelled). Any futures given by *fs* that
are duplicated will be returned once. Any futures that completed before
:func:`as_completed` is called will be yielded first. The returned iterator
raises a :exc:`concurrent.futures.TimeoutError` if :meth:`~iterator.__next__`
is called and the result isn't available after *timeout* seconds from the
original call to :func:`as_completed`. *timeout* can be an int or float. If
*timeout* is not specified or ``None``, there is no limit to the wait time.
.. seealso::
:pep:`3148` -- futures - execute computations asynchronously
The proposal which described this feature for inclusion in the Python
standard library.
Exception classes
-----------------
.. currentmodule:: concurrent.futures
.. exception:: CancelledError
Raised when a future is cancelled.
.. exception:: TimeoutError
Raised when a future operation exceeds the given timeout.
.. currentmodule:: concurrent.futures.process
.. exception:: BrokenProcessPool
Derived from :exc:`RuntimeError`, this exception class is raised when
one of the workers of a :class:`ProcessPoolExecutor` has terminated
in a non-clean fashion (for example, if it was killed from the outside).
.. versionadded:: 3.3

6
third_party/python/Doc/library/concurrent.rst vendored Normal file
View file

@ -0,0 +1,6 @@
The :mod:`concurrent` package
=============================
Currently, there is only one module in this package:
* :mod:`concurrent.futures` -- Launching parallel tasks

1324
third_party/python/Doc/library/configparser.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

100
third_party/python/Doc/library/constants.rst vendored Normal file
View file

@ -0,0 +1,100 @@
.. _built-in-consts:
Built-in Constants
==================
A small number of constants live in the built-in namespace. They are:
.. data:: False
The false value of the :class:`bool` type. Assignments to ``False``
are illegal and raise a :exc:`SyntaxError`.
.. data:: True
The true value of the :class:`bool` type. Assignments to ``True``
are illegal and raise a :exc:`SyntaxError`.
.. data:: None
The sole value of the type ``NoneType``. ``None`` is frequently used to
represent the absence of a value, as when default arguments are not passed to a
function. Assignments to ``None`` are illegal and raise a :exc:`SyntaxError`.
.. data:: NotImplemented
Special value which should be returned by the binary special methods
(e.g. :meth:`__eq__`, :meth:`__lt__`, :meth:`__add__`, :meth:`__rsub__`,
etc.) to indicate that the operation is not implemented with respect to
the other type; may be returned by the in-place binary special methods
(e.g. :meth:`__imul__`, :meth:`__iand__`, etc.) for the same purpose.
Its truth value is true.
.. note::
When a binary (or in-place) method returns ``NotImplemented`` the
interpreter will try the reflected operation on the other type (or some
other fallback, depending on the operator). If all attempts return
``NotImplemented``, the interpreter will raise an appropriate exception.
Incorrectly returning ``NotImplemented`` will result in a misleading
error message or the ``NotImplemented`` value being returned to Python code.
See :ref:`implementing-the-arithmetic-operations` for examples.
.. note::
``NotImplementedError`` and ``NotImplemented`` are not interchangeable,
even though they have similar names and purposes.
See :exc:`NotImplementedError` for details on when to use it.
.. index:: single: ...; ellipsis literal
.. data:: Ellipsis
The same as the ellipsis literal "``...``". Special value used mostly in conjunction
with extended slicing syntax for user-defined container data types.
.. data:: __debug__
This constant is true if Python was not started with an :option:`-O` option.
See also the :keyword:`assert` statement.
.. note::
The names :data:`None`, :data:`False`, :data:`True` and :data:`__debug__`
cannot be reassigned (assignments to them, even as an attribute name, raise
:exc:`SyntaxError`), so they can be considered "true" constants.
Constants added by the :mod:`site` module
-----------------------------------------
The :mod:`site` module (which is imported automatically during startup, except
if the :option:`-S` command-line option is given) adds several constants to the
built-in namespace. They are useful for the interactive interpreter shell and
should not be used in programs.
.. data:: quit(code=None)
exit(code=None)
Objects that when printed, print a message like "Use quit() or Ctrl-D
(i.e. EOF) to exit", and when called, raise :exc:`SystemExit` with the
specified exit code.
.. data:: copyright
credits
Objects that when printed or called, print the text of copyright or
credits, respectively.
.. data:: license
Object that when printed, prints the message "Type license() to see the
full license text", and when called, displays the full license text in a
pager-like fashion (one screen at a time).

777
third_party/python/Doc/library/contextlib.rst vendored Normal file
View file

@ -0,0 +1,777 @@
:mod:`contextlib` --- Utilities for :keyword:`with`\ -statement contexts
========================================================================
.. module:: contextlib
:synopsis: Utilities for with-statement contexts.
**Source code:** :source:`Lib/contextlib.py`
--------------
This module provides utilities for common tasks involving the :keyword:`with`
statement. For more information see also :ref:`typecontextmanager` and
:ref:`context-managers`.
Utilities
---------
Functions and classes provided:
.. class:: AbstractContextManager
An :term:`abstract base class` for classes that implement
:meth:`object.__enter__` and :meth:`object.__exit__`. A default
implementation for :meth:`object.__enter__` is provided which returns
``self`` while :meth:`object.__exit__` is an abstract method which by default
returns ``None``. See also the definition of :ref:`typecontextmanager`.
.. versionadded:: 3.6
.. decorator:: contextmanager
This function is a :term:`decorator` that can be used to define a factory
function for :keyword:`with` statement context managers, without needing to
create a class or separate :meth:`__enter__` and :meth:`__exit__` methods.
While many objects natively support use in with statements, sometimes a
resource needs to be managed that isn't a context manager in its own right,
and doesn't implement a ``close()`` method for use with ``contextlib.closing``
An abstract example would be the following to ensure correct resource
management::
from contextlib import contextmanager
@contextmanager
def managed_resource(*args, **kwds):
# Code to acquire resource, e.g.:
resource = acquire_resource(*args, **kwds)
try:
yield resource
finally:
# Code to release resource, e.g.:
release_resource(resource)
>>> with managed_resource(timeout=3600) as resource:
... # Resource is released at the end of this block,
... # even if code in the block raises an exception
The function being decorated must return a :term:`generator`-iterator when
called. This iterator must yield exactly one value, which will be bound to
the targets in the :keyword:`with` statement's :keyword:`as` clause, if any.
At the point where the generator yields, the block nested in the :keyword:`with`
statement is executed. The generator is then resumed after the block is exited.
If an unhandled exception occurs in the block, it is reraised inside the
generator at the point where the yield occurred. Thus, you can use a
:keyword:`try`...\ :keyword:`except`...\ :keyword:`finally` statement to trap
the error (if any), or ensure that some cleanup takes place. If an exception is
trapped merely in order to log it or to perform some action (rather than to
suppress it entirely), the generator must reraise that exception. Otherwise the
generator context manager will indicate to the :keyword:`with` statement that
the exception has been handled, and execution will resume with the statement
immediately following the :keyword:`with` statement.
:func:`contextmanager` uses :class:`ContextDecorator` so the context managers
it creates can be used as decorators as well as in :keyword:`with` statements.
When used as a decorator, a new generator instance is implicitly created on
each function call (this allows the otherwise "one-shot" context managers
created by :func:`contextmanager` to meet the requirement that context
managers support multiple invocations in order to be used as decorators).
.. versionchanged:: 3.2
Use of :class:`ContextDecorator`.
.. function:: closing(thing)
Return a context manager that closes *thing* upon completion of the block. This
is basically equivalent to::
from contextlib import contextmanager
@contextmanager
def closing(thing):
try:
yield thing
finally:
thing.close()
And lets you write code like this::
from contextlib import closing
from urllib.request import urlopen
with closing(urlopen('http://www.python.org')) as page:
for line in page:
print(line)
without needing to explicitly close ``page``. Even if an error occurs,
``page.close()`` will be called when the :keyword:`with` block is exited.
.. function:: suppress(*exceptions)
Return a context manager that suppresses any of the specified exceptions
if they occur in the body of a with statement and then resumes execution
with the first statement following the end of the with statement.
As with any other mechanism that completely suppresses exceptions, this
context manager should be used only to cover very specific errors where
silently continuing with program execution is known to be the right
thing to do.
For example::
from contextlib import suppress
with suppress(FileNotFoundError):
os.remove('somefile.tmp')
with suppress(FileNotFoundError):
os.remove('someotherfile.tmp')
This code is equivalent to::
try:
os.remove('somefile.tmp')
except FileNotFoundError:
pass
try:
os.remove('someotherfile.tmp')
except FileNotFoundError:
pass
This context manager is :ref:`reentrant <reentrant-cms>`.
.. versionadded:: 3.4
.. function:: redirect_stdout(new_target)
Context manager for temporarily redirecting :data:`sys.stdout` to
another file or file-like object.
This tool adds flexibility to existing functions or classes whose output
is hardwired to stdout.
For example, the output of :func:`help` normally is sent to *sys.stdout*.
You can capture that output in a string by redirecting the output to an
:class:`io.StringIO` object::
f = io.StringIO()
with redirect_stdout(f):
help(pow)
s = f.getvalue()
To send the output of :func:`help` to a file on disk, redirect the output
to a regular file::
with open('help.txt', 'w') as f:
with redirect_stdout(f):
help(pow)
To send the output of :func:`help` to *sys.stderr*::
with redirect_stdout(sys.stderr):
help(pow)
Note that the global side effect on :data:`sys.stdout` means that this
context manager is not suitable for use in library code and most threaded
applications. It also has no effect on the output of subprocesses.
However, it is still a useful approach for many utility scripts.
This context manager is :ref:`reentrant <reentrant-cms>`.
.. versionadded:: 3.4
.. function:: redirect_stderr(new_target)
Similar to :func:`~contextlib.redirect_stdout` but redirecting
:data:`sys.stderr` to another file or file-like object.
This context manager is :ref:`reentrant <reentrant-cms>`.
.. versionadded:: 3.5
.. class:: ContextDecorator()
A base class that enables a context manager to also be used as a decorator.
Context managers inheriting from ``ContextDecorator`` have to implement
``__enter__`` and ``__exit__`` as normal. ``__exit__`` retains its optional
exception handling even when used as a decorator.
``ContextDecorator`` is used by :func:`contextmanager`, so you get this
functionality automatically.
Example of ``ContextDecorator``::
from contextlib import ContextDecorator
class mycontext(ContextDecorator):
def __enter__(self):
print('Starting')
return self
def __exit__(self, *exc):
print('Finishing')
return False
>>> @mycontext()
... def function():
... print('The bit in the middle')
...
>>> function()
Starting
The bit in the middle
Finishing
>>> with mycontext():
... print('The bit in the middle')
...
Starting
The bit in the middle
Finishing
This change is just syntactic sugar for any construct of the following form::
def f():
with cm():
# Do stuff
``ContextDecorator`` lets you instead write::
@cm()
def f():
# Do stuff
It makes it clear that the ``cm`` applies to the whole function, rather than
just a piece of it (and saving an indentation level is nice, too).
Existing context managers that already have a base class can be extended by
using ``ContextDecorator`` as a mixin class::
from contextlib import ContextDecorator
class mycontext(ContextBaseClass, ContextDecorator):
def __enter__(self):
return self
def __exit__(self, *exc):
return False
.. note::
As the decorated function must be able to be called multiple times, the
underlying context manager must support use in multiple :keyword:`with`
statements. If this is not the case, then the original construct with the
explicit :keyword:`with` statement inside the function should be used.
.. versionadded:: 3.2
.. class:: ExitStack()
A context manager that is designed to make it easy to programmatically
combine other context managers and cleanup functions, especially those
that are optional or otherwise driven by input data.
For example, a set of files may easily be handled in a single with
statement as follows::
with ExitStack() as stack:
files = [stack.enter_context(open(fname)) for fname in filenames]
# All opened files will automatically be closed at the end of
# the with statement, even if attempts to open files later
# in the list raise an exception
Each instance maintains a stack of registered callbacks that are called in
reverse order when the instance is closed (either explicitly or implicitly
at the end of a :keyword:`with` statement). Note that callbacks are *not*
invoked implicitly when the context stack instance is garbage collected.
This stack model is used so that context managers that acquire their
resources in their ``__init__`` method (such as file objects) can be
handled correctly.
Since registered callbacks are invoked in the reverse order of
registration, this ends up behaving as if multiple nested :keyword:`with`
statements had been used with the registered set of callbacks. This even
extends to exception handling - if an inner callback suppresses or replaces
an exception, then outer callbacks will be passed arguments based on that
updated state.
This is a relatively low level API that takes care of the details of
correctly unwinding the stack of exit callbacks. It provides a suitable
foundation for higher level context managers that manipulate the exit
stack in application specific ways.
.. versionadded:: 3.3
.. method:: enter_context(cm)
Enters a new context manager and adds its :meth:`__exit__` method to
the callback stack. The return value is the result of the context
manager's own :meth:`__enter__` method.
These context managers may suppress exceptions just as they normally
would if used directly as part of a :keyword:`with` statement.
.. method:: push(exit)
Adds a context manager's :meth:`__exit__` method to the callback stack.
As ``__enter__`` is *not* invoked, this method can be used to cover
part of an :meth:`__enter__` implementation with a context manager's own
:meth:`__exit__` method.
If passed an object that is not a context manager, this method assumes
it is a callback with the same signature as a context manager's
:meth:`__exit__` method and adds it directly to the callback stack.
By returning true values, these callbacks can suppress exceptions the
same way context manager :meth:`__exit__` methods can.
The passed in object is returned from the function, allowing this
method to be used as a function decorator.
.. method:: callback(callback, *args, **kwds)
Accepts an arbitrary callback function and arguments and adds it to
the callback stack.
Unlike the other methods, callbacks added this way cannot suppress
exceptions (as they are never passed the exception details).
The passed in callback is returned from the function, allowing this
method to be used as a function decorator.
.. method:: pop_all()
Transfers the callback stack to a fresh :class:`ExitStack` instance
and returns it. No callbacks are invoked by this operation - instead,
they will now be invoked when the new stack is closed (either
explicitly or implicitly at the end of a :keyword:`with` statement).
For example, a group of files can be opened as an "all or nothing"
operation as follows::
with ExitStack() as stack:
files = [stack.enter_context(open(fname)) for fname in filenames]
# Hold onto the close method, but don't call it yet.
close_files = stack.pop_all().close
# If opening any file fails, all previously opened files will be
# closed automatically. If all files are opened successfully,
# they will remain open even after the with statement ends.
# close_files() can then be invoked explicitly to close them all.
.. method:: close()
Immediately unwinds the callback stack, invoking callbacks in the
reverse order of registration. For any context managers and exit
callbacks registered, the arguments passed in will indicate that no
exception occurred.
Examples and Recipes
--------------------
This section describes some examples and recipes for making effective use of
the tools provided by :mod:`contextlib`.
Supporting a variable number of context managers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The primary use case for :class:`ExitStack` is the one given in the class
documentation: supporting a variable number of context managers and other
cleanup operations in a single :keyword:`with` statement. The variability
may come from the number of context managers needed being driven by user
input (such as opening a user specified collection of files), or from
some of the context managers being optional::
with ExitStack() as stack:
for resource in resources:
stack.enter_context(resource)
if need_special_resource():
special = acquire_special_resource()
stack.callback(release_special_resource, special)
# Perform operations that use the acquired resources
As shown, :class:`ExitStack` also makes it quite easy to use :keyword:`with`
statements to manage arbitrary resources that don't natively support the
context management protocol.
Simplifying support for single optional context managers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In the specific case of a single optional context manager, :class:`ExitStack`
instances can be used as a "do nothing" context manager, allowing a context
manager to easily be omitted without affecting the overall structure of
the source code::
def debug_trace(details):
if __debug__:
return TraceContext(details)
# Don't do anything special with the context in release mode
return ExitStack()
with debug_trace():
# Suite is traced in debug mode, but runs normally otherwise
Catching exceptions from ``__enter__`` methods
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is occasionally desirable to catch exceptions from an ``__enter__``
method implementation, *without* inadvertently catching exceptions from
the :keyword:`with` statement body or the context manager's ``__exit__``
method. By using :class:`ExitStack` the steps in the context management
protocol can be separated slightly in order to allow this::
stack = ExitStack()
try:
x = stack.enter_context(cm)
except Exception:
# handle __enter__ exception
else:
with stack:
# Handle normal case
Actually needing to do this is likely to indicate that the underlying API
should be providing a direct resource management interface for use with
:keyword:`try`/:keyword:`except`/:keyword:`finally` statements, but not
all APIs are well designed in that regard. When a context manager is the
only resource management API provided, then :class:`ExitStack` can make it
easier to handle various situations that can't be handled directly in a
:keyword:`with` statement.
Cleaning up in an ``__enter__`` implementation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As noted in the documentation of :meth:`ExitStack.push`, this
method can be useful in cleaning up an already allocated resource if later
steps in the :meth:`__enter__` implementation fail.
Here's an example of doing this for a context manager that accepts resource
acquisition and release functions, along with an optional validation function,
and maps them to the context management protocol::
from contextlib import contextmanager, AbstractContextManager, ExitStack
class ResourceManager(AbstractContextManager):
def __init__(self, acquire_resource, release_resource, check_resource_ok=None):
self.acquire_resource = acquire_resource
self.release_resource = release_resource
if check_resource_ok is None:
def check_resource_ok(resource):
return True
self.check_resource_ok = check_resource_ok
@contextmanager
def _cleanup_on_error(self):
with ExitStack() as stack:
stack.push(self)
yield
# The validation check passed and didn't raise an exception
# Accordingly, we want to keep the resource, and pass it
# back to our caller
stack.pop_all()
def __enter__(self):
resource = self.acquire_resource()
with self._cleanup_on_error():
if not self.check_resource_ok(resource):
msg = "Failed validation for {!r}"
raise RuntimeError(msg.format(resource))
return resource
def __exit__(self, *exc_details):
# We don't need to duplicate any of our resource release logic
self.release_resource()
Replacing any use of ``try-finally`` and flag variables
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A pattern you will sometimes see is a ``try-finally`` statement with a flag
variable to indicate whether or not the body of the ``finally`` clause should
be executed. In its simplest form (that can't already be handled just by
using an ``except`` clause instead), it looks something like this::
cleanup_needed = True
try:
result = perform_operation()
if result:
cleanup_needed = False
finally:
if cleanup_needed:
cleanup_resources()
As with any ``try`` statement based code, this can cause problems for
development and review, because the setup code and the cleanup code can end
up being separated by arbitrarily long sections of code.
:class:`ExitStack` makes it possible to instead register a callback for
execution at the end of a ``with`` statement, and then later decide to skip
executing that callback::
from contextlib import ExitStack
with ExitStack() as stack:
stack.callback(cleanup_resources)
result = perform_operation()
if result:
stack.pop_all()
This allows the intended cleanup up behaviour to be made explicit up front,
rather than requiring a separate flag variable.
If a particular application uses this pattern a lot, it can be simplified
even further by means of a small helper class::
from contextlib import ExitStack
class Callback(ExitStack):
def __init__(self, callback, *args, **kwds):
super(Callback, self).__init__()
self.callback(callback, *args, **kwds)
def cancel(self):
self.pop_all()
with Callback(cleanup_resources) as cb:
result = perform_operation()
if result:
cb.cancel()
If the resource cleanup isn't already neatly bundled into a standalone
function, then it is still possible to use the decorator form of
:meth:`ExitStack.callback` to declare the resource cleanup in
advance::
from contextlib import ExitStack
with ExitStack() as stack:
@stack.callback
def cleanup_resources():
...
result = perform_operation()
if result:
stack.pop_all()
Due to the way the decorator protocol works, a callback function
declared this way cannot take any parameters. Instead, any resources to
be released must be accessed as closure variables.
Using a context manager as a function decorator
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:class:`ContextDecorator` makes it possible to use a context manager in
both an ordinary ``with`` statement and also as a function decorator.
For example, it is sometimes useful to wrap functions or groups of statements
with a logger that can track the time of entry and time of exit. Rather than
writing both a function decorator and a context manager for the task,
inheriting from :class:`ContextDecorator` provides both capabilities in a
single definition::
from contextlib import ContextDecorator
import logging
logging.basicConfig(level=logging.INFO)
class track_entry_and_exit(ContextDecorator):
def __init__(self, name):
self.name = name
def __enter__(self):
logging.info('Entering: %s', self.name)
def __exit__(self, exc_type, exc, exc_tb):
logging.info('Exiting: %s', self.name)
Instances of this class can be used as both a context manager::
with track_entry_and_exit('widget loader'):
print('Some time consuming activity goes here')
load_widget()
And also as a function decorator::
@track_entry_and_exit('widget loader')
def activity():
print('Some time consuming activity goes here')
load_widget()
Note that there is one additional limitation when using context managers
as function decorators: there's no way to access the return value of
:meth:`__enter__`. If that value is needed, then it is still necessary to use
an explicit ``with`` statement.
.. seealso::
:pep:`343` - The "with" statement
The specification, background, and examples for the Python :keyword:`with`
statement.
.. _single-use-reusable-and-reentrant-cms:
Single use, reusable and reentrant context managers
---------------------------------------------------
Most context managers are written in a way that means they can only be
used effectively in a :keyword:`with` statement once. These single use
context managers must be created afresh each time they're used -
attempting to use them a second time will trigger an exception or
otherwise not work correctly.
This common limitation means that it is generally advisable to create
context managers directly in the header of the :keyword:`with` statement
where they are used (as shown in all of the usage examples above).
Files are an example of effectively single use context managers, since
the first :keyword:`with` statement will close the file, preventing any
further IO operations using that file object.
Context managers created using :func:`contextmanager` are also single use
context managers, and will complain about the underlying generator failing
to yield if an attempt is made to use them a second time::
>>> from contextlib import contextmanager
>>> @contextmanager
... def singleuse():
... print("Before")
... yield
... print("After")
...
>>> cm = singleuse()
>>> with cm:
... pass
...
Before
After
>>> with cm:
... pass
...
Traceback (most recent call last):
...
RuntimeError: generator didn't yield
.. _reentrant-cms:
Reentrant context managers
^^^^^^^^^^^^^^^^^^^^^^^^^^
More sophisticated context managers may be "reentrant". These context
managers can not only be used in multiple :keyword:`with` statements,
but may also be used *inside* a :keyword:`with` statement that is already
using the same context manager.
:class:`threading.RLock` is an example of a reentrant context manager, as are
:func:`suppress` and :func:`redirect_stdout`. Here's a very simple example of
reentrant use::
>>> from contextlib import redirect_stdout
>>> from io import StringIO
>>> stream = StringIO()
>>> write_to_stream = redirect_stdout(stream)
>>> with write_to_stream:
... print("This is written to the stream rather than stdout")
... with write_to_stream:
... print("This is also written to the stream")
...
>>> print("This is written directly to stdout")
This is written directly to stdout
>>> print(stream.getvalue())
This is written to the stream rather than stdout
This is also written to the stream
Real world examples of reentrancy are more likely to involve multiple
functions calling each other and hence be far more complicated than this
example.
Note also that being reentrant is *not* the same thing as being thread safe.
:func:`redirect_stdout`, for example, is definitely not thread safe, as it
makes a global modification to the system state by binding :data:`sys.stdout`
to a different stream.
.. _reusable-cms:
Reusable context managers
^^^^^^^^^^^^^^^^^^^^^^^^^
Distinct from both single use and reentrant context managers are "reusable"
context managers (or, to be completely explicit, "reusable, but not
reentrant" context managers, since reentrant context managers are also
reusable). These context managers support being used multiple times, but
will fail (or otherwise not work correctly) if the specific context manager
instance has already been used in a containing with statement.
:class:`threading.Lock` is an example of a reusable, but not reentrant,
context manager (for a reentrant lock, it is necessary to use
:class:`threading.RLock` instead).
Another example of a reusable, but not reentrant, context manager is
:class:`ExitStack`, as it invokes *all* currently registered callbacks
when leaving any with statement, regardless of where those callbacks
were added::
>>> from contextlib import ExitStack
>>> stack = ExitStack()
>>> with stack:
... stack.callback(print, "Callback: from first context")
... print("Leaving first context")
...
Leaving first context
Callback: from first context
>>> with stack:
... stack.callback(print, "Callback: from second context")
... print("Leaving second context")
...
Leaving second context
Callback: from second context
>>> with stack:
... stack.callback(print, "Callback: from outer context")
... with stack:
... stack.callback(print, "Callback: from inner context")
... print("Leaving inner context")
... print("Leaving outer context")
...
Leaving inner context
Callback: from inner context
Callback: from outer context
Leaving outer context
As the output from the example shows, reusing a single stack object across
multiple with statements works correctly, but attempting to nest them
will cause the stack to be cleared at the end of the innermost with
statement, which is unlikely to be desirable behaviour.
Using separate :class:`ExitStack` instances instead of reusing a single
instance avoids that problem::
>>> from contextlib import ExitStack
>>> with ExitStack() as outer_stack:
... outer_stack.callback(print, "Callback: from outer context")
... with ExitStack() as inner_stack:
... inner_stack.callback(print, "Callback: from inner context")
... print("Leaving inner context")
... print("Leaving outer context")
...
Leaving inner context
Callback: from inner context
Leaving outer context
Callback: from outer context

95
third_party/python/Doc/library/copy.rst vendored Normal file
View file

@ -0,0 +1,95 @@
:mod:`copy` --- Shallow and deep copy operations
================================================
.. module:: copy
:synopsis: Shallow and deep copy operations.
**Source code:** :source:`Lib/copy.py`
--------------
Assignment statements in Python do not copy objects, they create bindings
between a target and an object. For collections that are mutable or contain
mutable items, a copy is sometimes needed so one can change one copy without
changing the other. This module provides generic shallow and deep copy
operations (explained below).
Interface summary:
.. function:: copy(x)
Return a shallow copy of *x*.
.. function:: deepcopy(x[, memo])
Return a deep copy of *x*.
.. exception:: error
Raised for module specific errors.
The difference between shallow and deep copying is only relevant for compound
objects (objects that contain other objects, like lists or class instances):
* A *shallow copy* constructs a new compound object and then (to the extent
possible) inserts *references* into it to the objects found in the original.
* A *deep copy* constructs a new compound object and then, recursively, inserts
*copies* into it of the objects found in the original.
Two problems often exist with deep copy operations that don't exist with shallow
copy operations:
* Recursive objects (compound objects that, directly or indirectly, contain a
reference to themselves) may cause a recursive loop.
* Because deep copy copies everything it may copy too much, such as data
which is intended to be shared between copies.
The :func:`deepcopy` function avoids these problems by:
* keeping a ``memo`` dictionary of objects already copied during the current
copying pass; and
* letting user-defined classes override the copying operation or the set of
components copied.
This module does not copy types like module, method, stack trace, stack frame,
file, socket, window, array, or any similar types. It does "copy" functions and
classes (shallow and deeply), by returning the original object unchanged; this
is compatible with the way these are treated by the :mod:`pickle` module.
Shallow copies of dictionaries can be made using :meth:`dict.copy`, and
of lists by assigning a slice of the entire list, for example,
``copied_list = original_list[:]``.
.. index:: module: pickle
Classes can use the same interfaces to control copying that they use to control
pickling. See the description of module :mod:`pickle` for information on these
methods. In fact, the :mod:`copy` module uses the registered
pickle functions from the :mod:`copyreg` module.
.. index::
single: __copy__() (copy protocol)
single: __deepcopy__() (copy protocol)
In order for a class to define its own copy implementation, it can define
special methods :meth:`__copy__` and :meth:`__deepcopy__`. The former is called
to implement the shallow copy operation; no additional arguments are passed.
The latter is called to implement the deep copy operation; it is passed one
argument, the ``memo`` dictionary. If the :meth:`__deepcopy__` implementation needs
to make a deep copy of a component, it should call the :func:`deepcopy` function
with the component as first argument and the memo dictionary as second argument.
.. seealso::
Module :mod:`pickle`
Discussion of the special methods used to support object state retrieval and
restoration.

65
third_party/python/Doc/library/copyreg.rst vendored Normal file
View file

@ -0,0 +1,65 @@
:mod:`copyreg` --- Register :mod:`pickle` support functions
===========================================================
.. module:: copyreg
:synopsis: Register pickle support functions.
**Source code:** :source:`Lib/copyreg.py`
.. index::
module: pickle
module: copy
--------------
The :mod:`copyreg` module offers a way to define functions used while pickling
specific objects. The :mod:`pickle` and :mod:`copy` modules use those functions
when pickling/copying those objects. The module provides configuration
information about object constructors which are not classes.
Such constructors may be factory functions or class instances.
.. function:: constructor(object)
Declares *object* to be a valid constructor. If *object* is not callable (and
hence not valid as a constructor), raises :exc:`TypeError`.
.. function:: pickle(type, function, constructor=None)
Declares that *function* should be used as a "reduction" function for objects
of type *type*. *function* should return either a string or a tuple
containing two or three elements.
The optional *constructor* parameter, if provided, is a callable object which
can be used to reconstruct the object when called with the tuple of arguments
returned by *function* at pickling time. :exc:`TypeError` will be raised if
*object* is a class or *constructor* is not callable.
See the :mod:`pickle` module for more details on the interface
expected of *function* and *constructor*. Note that the
:attr:`~pickle.Pickler.dispatch_table` attribute of a pickler
object or subclass of :class:`pickle.Pickler` can also be used for
declaring reduction functions.
Example
-------
The example below would like to show how to register a pickle function and how
it will be used:
>>> import copyreg, copy, pickle
>>> class C(object):
... def __init__(self, a):
... self.a = a
...
>>> def pickle_c(c):
... print("pickling a C instance...")
... return C, (c.a,)
...
>>> copyreg.pickle(C, pickle_c)
>>> c = C(1)
>>> d = copy.copy(c)
pickling a C instance...
>>> p = pickle.dumps(c)
pickling a C instance...

156
third_party/python/Doc/library/crypt.rst vendored Normal file
View file

@ -0,0 +1,156 @@
:mod:`crypt` --- Function to check Unix passwords
=================================================
.. module:: crypt
:platform: Unix
:synopsis: The crypt() function used to check Unix passwords.
.. moduleauthor:: Steven D. Majewski <sdm7g@virginia.edu>
.. sectionauthor:: Steven D. Majewski <sdm7g@virginia.edu>
.. sectionauthor:: Peter Funk <pf@artcom-gmbh.de>
**Source code:** :source:`Lib/crypt.py`
.. index::
single: crypt(3)
pair: cipher; DES
--------------
This module implements an interface to the :manpage:`crypt(3)` routine, which is
a one-way hash function based upon a modified DES algorithm; see the Unix man
page for further details. Possible uses include storing hashed passwords
so you can check passwords without storing the actual password, or attempting
to crack Unix passwords with a dictionary.
.. index:: single: crypt(3)
Notice that the behavior of this module depends on the actual implementation of
the :manpage:`crypt(3)` routine in the running system. Therefore, any
extensions available on the current implementation will also be available on
this module.
Hashing Methods
---------------
.. versionadded:: 3.3
The :mod:`crypt` module defines the list of hashing methods (not all methods
are available on all platforms):
.. data:: METHOD_SHA512
A Modular Crypt Format method with 16 character salt and 86 character
hash. This is the strongest method.
.. data:: METHOD_SHA256
Another Modular Crypt Format method with 16 character salt and 43
character hash.
.. data:: METHOD_MD5
Another Modular Crypt Format method with 8 character salt and 22
character hash.
.. data:: METHOD_CRYPT
The traditional method with a 2 character salt and 13 characters of
hash. This is the weakest method.
Module Attributes
-----------------
.. versionadded:: 3.3
.. attribute:: methods
A list of available password hashing algorithms, as
``crypt.METHOD_*`` objects. This list is sorted from strongest to
weakest.
Module Functions
----------------
The :mod:`crypt` module defines the following functions:
.. function:: crypt(word, salt=None)
*word* will usually be a user's password as typed at a prompt or in a graphical
interface. The optional *salt* is either a string as returned from
:func:`mksalt`, one of the ``crypt.METHOD_*`` values (though not all
may be available on all platforms), or a full encrypted password
including salt, as returned by this function. If *salt* is not
provided, the strongest method will be used (as returned by
:func:`methods`).
Checking a password is usually done by passing the plain-text password
as *word* and the full results of a previous :func:`crypt` call,
which should be the same as the results of this call.
*salt* (either a random 2 or 16 character string, possibly prefixed with
``$digit$`` to indicate the method) which will be used to perturb the
encryption algorithm. The characters in *salt* must be in the set
``[./a-zA-Z0-9]``, with the exception of Modular Crypt Format which
prefixes a ``$digit$``.
Returns the hashed password as a string, which will be composed of
characters from the same alphabet as the salt.
.. index:: single: crypt(3)
Since a few :manpage:`crypt(3)` extensions allow different values, with
different sizes in the *salt*, it is recommended to use the full crypted
password as salt when checking for a password.
.. versionchanged:: 3.3
Accept ``crypt.METHOD_*`` values in addition to strings for *salt*.
.. function:: mksalt(method=None)
Return a randomly generated salt of the specified method. If no
*method* is given, the strongest method available as returned by
:func:`methods` is used.
The return value is a string either of 2 characters in length for
``crypt.METHOD_CRYPT``, or 19 characters starting with ``$digit$`` and
16 random characters from the set ``[./a-zA-Z0-9]``, suitable for
passing as the *salt* argument to :func:`crypt`.
.. versionadded:: 3.3
Examples
--------
A simple example illustrating typical use (a constant-time comparison
operation is needed to limit exposure to timing attacks.
:func:`hmac.compare_digest` is suitable for this purpose)::
import pwd
import crypt
import getpass
from hmac import compare_digest as compare_hash
def login():
username = input('Python login: ')
cryptedpasswd = pwd.getpwnam(username)[1]
if cryptedpasswd:
if cryptedpasswd == 'x' or cryptedpasswd == '*':
raise ValueError('no support for shadow passwords')
cleartext = getpass.getpass()
return compare_hash(crypt.crypt(cleartext, cryptedpasswd), cryptedpasswd)
else:
return True
To generate a hash of a password using the strongest available method and
check it against the original::
import crypt
from hmac import compare_digest as compare_hash
hashed = crypt.crypt(plaintext)
if not compare_hash(hashed, crypt.crypt(plaintext, hashed)):
raise ValueError("hashed version doesn't validate against original")

19
third_party/python/Doc/library/crypto.rst vendored Normal file
View file

@ -0,0 +1,19 @@
.. _crypto:
**********************
Cryptographic Services
**********************
.. index:: single: cryptography
The modules described in this chapter implement various algorithms of a
cryptographic nature. They are available at the discretion of the installation.
On Unix systems, the :mod:`crypt` module may also be available.
Here's an overview:
.. toctree::
hashlib.rst
hmac.rst
secrets.rst

552
third_party/python/Doc/library/csv.rst vendored Normal file
View file

@ -0,0 +1,552 @@
:mod:`csv` --- CSV File Reading and Writing
===========================================
.. module:: csv
:synopsis: Write and read tabular data to and from delimited files.
.. sectionauthor:: Skip Montanaro <skip@pobox.com>
**Source code:** :source:`Lib/csv.py`
.. index::
single: csv
pair: data; tabular
--------------
The so-called CSV (Comma Separated Values) format is the most common import and
export format for spreadsheets and databases. CSV format was used for many
years prior to attempts to describe the format in a standardized way in
:rfc:`4180`. The lack of a well-defined standard means that subtle differences
often exist in the data produced and consumed by different applications. These
differences can make it annoying to process CSV files from multiple sources.
Still, while the delimiters and quoting characters vary, the overall format is
similar enough that it is possible to write a single module which can
efficiently manipulate such data, hiding the details of reading and writing the
data from the programmer.
The :mod:`csv` module implements classes to read and write tabular data in CSV
format. It allows programmers to say, "write this data in the format preferred
by Excel," or "read data from this file which was generated by Excel," without
knowing the precise details of the CSV format used by Excel. Programmers can
also describe the CSV formats understood by other applications or define their
own special-purpose CSV formats.
The :mod:`csv` module's :class:`reader` and :class:`writer` objects read and
write sequences. Programmers can also read and write data in dictionary form
using the :class:`DictReader` and :class:`DictWriter` classes.
.. seealso::
:pep:`305` - CSV File API
The Python Enhancement Proposal which proposed this addition to Python.
.. _csv-contents:
Module Contents
---------------
The :mod:`csv` module defines the following functions:
.. index::
single: universal newlines; csv.reader function
.. function:: reader(csvfile, dialect='excel', **fmtparams)
Return a reader object which will iterate over lines in the given *csvfile*.
*csvfile* can be any object which supports the :term:`iterator` protocol and returns a
string each time its :meth:`!__next__` method is called --- :term:`file objects
<file object>` and list objects are both suitable. If *csvfile* is a file object,
it should be opened with ``newline=''``. [1]_ An optional
*dialect* parameter can be given which is used to define a set of parameters
specific to a particular CSV dialect. It may be an instance of a subclass of
the :class:`Dialect` class or one of the strings returned by the
:func:`list_dialects` function. The other optional *fmtparams* keyword arguments
can be given to override individual formatting parameters in the current
dialect. For full details about the dialect and formatting parameters, see
section :ref:`csv-fmt-params`.
Each row read from the csv file is returned as a list of strings. No
automatic data type conversion is performed unless the ``QUOTE_NONNUMERIC`` format
option is specified (in which case unquoted fields are transformed into floats).
A short usage example::
>>> import csv
>>> with open('eggs.csv', newline='') as csvfile:
... spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
... for row in spamreader:
... print(', '.join(row))
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
.. function:: writer(csvfile, dialect='excel', **fmtparams)
Return a writer object responsible for converting the user's data into delimited
strings on the given file-like object. *csvfile* can be any object with a
:func:`write` method. If *csvfile* is a file object, it should be opened with
``newline=''`` [1]_. An optional *dialect*
parameter can be given which is used to define a set of parameters specific to a
particular CSV dialect. It may be an instance of a subclass of the
:class:`Dialect` class or one of the strings returned by the
:func:`list_dialects` function. The other optional *fmtparams* keyword arguments
can be given to override individual formatting parameters in the current
dialect. For full details about the dialect and formatting parameters, see
section :ref:`csv-fmt-params`. To make it
as easy as possible to interface with modules which implement the DB API, the
value :const:`None` is written as the empty string. While this isn't a
reversible transformation, it makes it easier to dump SQL NULL data values to
CSV files without preprocessing the data returned from a ``cursor.fetch*`` call.
All other non-string data are stringified with :func:`str` before being written.
A short usage example::
import csv
with open('eggs.csv', 'w', newline='') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=' ',
quotechar='|', quoting=csv.QUOTE_MINIMAL)
spamwriter.writerow(['Spam'] * 5 + ['Baked Beans'])
spamwriter.writerow(['Spam', 'Lovely Spam', 'Wonderful Spam'])
.. function:: register_dialect(name[, dialect[, **fmtparams]])
Associate *dialect* with *name*. *name* must be a string. The
dialect can be specified either by passing a sub-class of :class:`Dialect`, or
by *fmtparams* keyword arguments, or both, with keyword arguments overriding
parameters of the dialect. For full details about the dialect and formatting
parameters, see section :ref:`csv-fmt-params`.
.. function:: unregister_dialect(name)
Delete the dialect associated with *name* from the dialect registry. An
:exc:`Error` is raised if *name* is not a registered dialect name.
.. function:: get_dialect(name)
Return the dialect associated with *name*. An :exc:`Error` is raised if
*name* is not a registered dialect name. This function returns an immutable
:class:`Dialect`.
.. function:: list_dialects()
Return the names of all registered dialects.
.. function:: field_size_limit([new_limit])
Returns the current maximum field size allowed by the parser. If *new_limit* is
given, this becomes the new limit.
The :mod:`csv` module defines the following classes:
.. class:: DictReader(f, fieldnames=None, restkey=None, restval=None, \
dialect='excel', *args, **kwds)
Create an object that operates like a regular reader but maps the
information in each row to an :mod:`OrderedDict <collections.OrderedDict>`
whose keys are given by the optional *fieldnames* parameter.
The *fieldnames* parameter is a :term:`sequence`. If *fieldnames* is
omitted, the values in the first row of file *f* will be used as the
fieldnames. Regardless of how the fieldnames are determined, the ordered
dictionary preserves their original ordering.
If a row has more fields than fieldnames, the remaining data is put in a
list and stored with the fieldname specified by *restkey* (which defaults
to ``None``). If a non-blank row has fewer fields than fieldnames, the
missing values are filled-in with ``None``.
All other optional or keyword arguments are passed to the underlying
:class:`reader` instance.
.. versionchanged:: 3.6
Returned rows are now of type :class:`OrderedDict`.
A short usage example::
>>> import csv
>>> with open('names.csv', newline='') as csvfile:
... reader = csv.DictReader(csvfile)
... for row in reader:
... print(row['first_name'], row['last_name'])
...
Eric Idle
John Cleese
>>> print(row)
OrderedDict([('first_name', 'John'), ('last_name', 'Cleese')])
.. class:: DictWriter(f, fieldnames, restval='', extrasaction='raise', \
dialect='excel', *args, **kwds)
Create an object which operates like a regular writer but maps dictionaries
onto output rows. The *fieldnames* parameter is a :mod:`sequence
<collections.abc>` of keys that identify the order in which values in the
dictionary passed to the :meth:`writerow` method are written to file
*f*. The optional *restval* parameter specifies the value to be
written if the dictionary is missing a key in *fieldnames*. If the
dictionary passed to the :meth:`writerow` method contains a key not found in
*fieldnames*, the optional *extrasaction* parameter indicates what action to
take.
If it is set to ``'raise'``, the default value, a :exc:`ValueError`
is raised.
If it is set to ``'ignore'``, extra values in the dictionary are ignored.
Any other optional or keyword arguments are passed to the underlying
:class:`writer` instance.
Note that unlike the :class:`DictReader` class, the *fieldnames* parameter
of the :class:`DictWriter` is not optional. Since Python's :class:`dict`
objects are not ordered, there is not enough information available to deduce
the order in which the row should be written to file *f*.
A short usage example::
import csv
with open('names.csv', 'w', newline='') as csvfile:
fieldnames = ['first_name', 'last_name']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
writer.writerow({'first_name': 'Baked', 'last_name': 'Beans'})
writer.writerow({'first_name': 'Lovely', 'last_name': 'Spam'})
writer.writerow({'first_name': 'Wonderful', 'last_name': 'Spam'})
.. class:: Dialect
The :class:`Dialect` class is a container class relied on primarily for its
attributes, which are used to define the parameters for a specific
:class:`reader` or :class:`writer` instance.
.. class:: excel()
The :class:`excel` class defines the usual properties of an Excel-generated CSV
file. It is registered with the dialect name ``'excel'``.
.. class:: excel_tab()
The :class:`excel_tab` class defines the usual properties of an Excel-generated
TAB-delimited file. It is registered with the dialect name ``'excel-tab'``.
.. class:: unix_dialect()
The :class:`unix_dialect` class defines the usual properties of a CSV file
generated on UNIX systems, i.e. using ``'\n'`` as line terminator and quoting
all fields. It is registered with the dialect name ``'unix'``.
.. versionadded:: 3.2
.. class:: Sniffer()
The :class:`Sniffer` class is used to deduce the format of a CSV file.
The :class:`Sniffer` class provides two methods:
.. method:: sniff(sample, delimiters=None)
Analyze the given *sample* and return a :class:`Dialect` subclass
reflecting the parameters found. If the optional *delimiters* parameter
is given, it is interpreted as a string containing possible valid
delimiter characters.
.. method:: has_header(sample)
Analyze the sample text (presumed to be in CSV format) and return
:const:`True` if the first row appears to be a series of column headers.
An example for :class:`Sniffer` use::
with open('example.csv', newline='') as csvfile:
dialect = csv.Sniffer().sniff(csvfile.read(1024))
csvfile.seek(0)
reader = csv.reader(csvfile, dialect)
# ... process CSV file contents here ...
The :mod:`csv` module defines the following constants:
.. data:: QUOTE_ALL
Instructs :class:`writer` objects to quote all fields.
.. data:: QUOTE_MINIMAL
Instructs :class:`writer` objects to only quote those fields which contain
special characters such as *delimiter*, *quotechar* or any of the characters in
*lineterminator*.
.. data:: QUOTE_NONNUMERIC
Instructs :class:`writer` objects to quote all non-numeric fields.
Instructs the reader to convert all non-quoted fields to type *float*.
.. data:: QUOTE_NONE
Instructs :class:`writer` objects to never quote fields. When the current
*delimiter* occurs in output data it is preceded by the current *escapechar*
character. If *escapechar* is not set, the writer will raise :exc:`Error` if
any characters that require escaping are encountered.
Instructs :class:`reader` to perform no special processing of quote characters.
The :mod:`csv` module defines the following exception:
.. exception:: Error
Raised by any of the functions when an error is detected.
.. _csv-fmt-params:
Dialects and Formatting Parameters
----------------------------------
To make it easier to specify the format of input and output records, specific
formatting parameters are grouped together into dialects. A dialect is a
subclass of the :class:`Dialect` class having a set of specific methods and a
single :meth:`validate` method. When creating :class:`reader` or
:class:`writer` objects, the programmer can specify a string or a subclass of
the :class:`Dialect` class as the dialect parameter. In addition to, or instead
of, the *dialect* parameter, the programmer can also specify individual
formatting parameters, which have the same names as the attributes defined below
for the :class:`Dialect` class.
Dialects support the following attributes:
.. attribute:: Dialect.delimiter
A one-character string used to separate fields. It defaults to ``','``.
.. attribute:: Dialect.doublequote
Controls how instances of *quotechar* appearing inside a field should
themselves be quoted. When :const:`True`, the character is doubled. When
:const:`False`, the *escapechar* is used as a prefix to the *quotechar*. It
defaults to :const:`True`.
On output, if *doublequote* is :const:`False` and no *escapechar* is set,
:exc:`Error` is raised if a *quotechar* is found in a field.
.. attribute:: Dialect.escapechar
A one-character string used by the writer to escape the *delimiter* if *quoting*
is set to :const:`QUOTE_NONE` and the *quotechar* if *doublequote* is
:const:`False`. On reading, the *escapechar* removes any special meaning from
the following character. It defaults to :const:`None`, which disables escaping.
.. attribute:: Dialect.lineterminator
The string used to terminate lines produced by the :class:`writer`. It defaults
to ``'\r\n'``.
.. note::
The :class:`reader` is hard-coded to recognise either ``'\r'`` or ``'\n'`` as
end-of-line, and ignores *lineterminator*. This behavior may change in the
future.
.. attribute:: Dialect.quotechar
A one-character string used to quote fields containing special characters, such
as the *delimiter* or *quotechar*, or which contain new-line characters. It
defaults to ``'"'``.
.. attribute:: Dialect.quoting
Controls when quotes should be generated by the writer and recognised by the
reader. It can take on any of the :const:`QUOTE_\*` constants (see section
:ref:`csv-contents`) and defaults to :const:`QUOTE_MINIMAL`.
.. attribute:: Dialect.skipinitialspace
When :const:`True`, whitespace immediately following the *delimiter* is ignored.
The default is :const:`False`.
.. attribute:: Dialect.strict
When ``True``, raise exception :exc:`Error` on bad CSV input.
The default is ``False``.
Reader Objects
--------------
Reader objects (:class:`DictReader` instances and objects returned by the
:func:`reader` function) have the following public methods:
.. method:: csvreader.__next__()
Return the next row of the reader's iterable object as a list (if the object
was returned from :func:`reader`) or a dict (if it is a :class:`DictReader`
instance), parsed according to the current dialect. Usually you should call
this as ``next(reader)``.
Reader objects have the following public attributes:
.. attribute:: csvreader.dialect
A read-only description of the dialect in use by the parser.
.. attribute:: csvreader.line_num
The number of lines read from the source iterator. This is not the same as the
number of records returned, as records can span multiple lines.
DictReader objects have the following public attribute:
.. attribute:: csvreader.fieldnames
If not passed as a parameter when creating the object, this attribute is
initialized upon first access or when the first record is read from the
file.
Writer Objects
--------------
:class:`Writer` objects (:class:`DictWriter` instances and objects returned by
the :func:`writer` function) have the following public methods. A *row* must be
an iterable of strings or numbers for :class:`Writer` objects and a dictionary
mapping fieldnames to strings or numbers (by passing them through :func:`str`
first) for :class:`DictWriter` objects. Note that complex numbers are written
out surrounded by parens. This may cause some problems for other programs which
read CSV files (assuming they support complex numbers at all).
.. method:: csvwriter.writerow(row)
Write the *row* parameter to the writer's file object, formatted according to
the current dialect.
.. versionchanged:: 3.5
Added support of arbitrary iterables.
.. method:: csvwriter.writerows(rows)
Write all elements in *rows* (an iterable of *row* objects as described
above) to the writer's file object, formatted according to the current
dialect.
Writer objects have the following public attribute:
.. attribute:: csvwriter.dialect
A read-only description of the dialect in use by the writer.
DictWriter objects have the following public method:
.. method:: DictWriter.writeheader()
Write a row with the field names (as specified in the constructor).
.. versionadded:: 3.2
.. _csv-examples:
Examples
--------
The simplest example of reading a CSV file::
import csv
with open('some.csv', newline='') as f:
reader = csv.reader(f)
for row in reader:
print(row)
Reading a file with an alternate format::
import csv
with open('passwd', newline='') as f:
reader = csv.reader(f, delimiter=':', quoting=csv.QUOTE_NONE)
for row in reader:
print(row)
The corresponding simplest possible writing example is::
import csv
with open('some.csv', 'w', newline='') as f:
writer = csv.writer(f)
writer.writerows(someiterable)
Since :func:`open` is used to open a CSV file for reading, the file
will by default be decoded into unicode using the system default
encoding (see :func:`locale.getpreferredencoding`). To decode a file
using a different encoding, use the ``encoding`` argument of open::
import csv
with open('some.csv', newline='', encoding='utf-8') as f:
reader = csv.reader(f)
for row in reader:
print(row)
The same applies to writing in something other than the system default
encoding: specify the encoding argument when opening the output file.
Registering a new dialect::
import csv
csv.register_dialect('unixpwd', delimiter=':', quoting=csv.QUOTE_NONE)
with open('passwd', newline='') as f:
reader = csv.reader(f, 'unixpwd')
A slightly more advanced use of the reader --- catching and reporting errors::
import csv, sys
filename = 'some.csv'
with open(filename, newline='') as f:
reader = csv.reader(f)
try:
for row in reader:
print(row)
except csv.Error as e:
sys.exit('file {}, line {}: {}'.format(filename, reader.line_num, e))
And while the module doesn't directly support parsing strings, it can easily be
done::
import csv
for row in csv.reader(['one,two,three']):
print(row)
.. rubric:: Footnotes
.. [1] If ``newline=''`` is not specified, newlines embedded inside quoted fields
will not be interpreted correctly, and on platforms that use ``\r\n`` linendings
on write an extra ``\r`` will be added. It should always be safe to specify
``newline=''``, since the csv module does its own
(:term:`universal <universal newlines>`) newline handling.

2484
third_party/python/Doc/library/ctypes.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

229
third_party/python/Doc/library/curses.ascii.rst vendored Normal file
View file

@ -0,0 +1,229 @@
:mod:`curses.ascii` --- Utilities for ASCII characters
======================================================
.. module:: curses.ascii
:synopsis: Constants and set-membership functions for ASCII characters.
.. moduleauthor:: Eric S. Raymond <esr@thyrsus.com>
.. sectionauthor:: Eric S. Raymond <esr@thyrsus.com>
--------------
The :mod:`curses.ascii` module supplies name constants for ASCII characters and
functions to test membership in various ASCII character classes. The constants
supplied are names for control characters as follows:
+--------------+----------------------------------------------+
| Name | Meaning |
+==============+==============================================+
| :const:`NUL` | |
+--------------+----------------------------------------------+
| :const:`SOH` | Start of heading, console interrupt |
+--------------+----------------------------------------------+
| :const:`STX` | Start of text |
+--------------+----------------------------------------------+
| :const:`ETX` | End of text |
+--------------+----------------------------------------------+
| :const:`EOT` | End of transmission |
+--------------+----------------------------------------------+
| :const:`ENQ` | Enquiry, goes with :const:`ACK` flow control |
+--------------+----------------------------------------------+
| :const:`ACK` | Acknowledgement |
+--------------+----------------------------------------------+
| :const:`BEL` | Bell |
+--------------+----------------------------------------------+
| :const:`BS` | Backspace |
+--------------+----------------------------------------------+
| :const:`TAB` | Tab |
+--------------+----------------------------------------------+
| :const:`HT` | Alias for :const:`TAB`: "Horizontal tab" |
+--------------+----------------------------------------------+
| :const:`LF` | Line feed |
+--------------+----------------------------------------------+
| :const:`NL` | Alias for :const:`LF`: "New line" |
+--------------+----------------------------------------------+
| :const:`VT` | Vertical tab |
+--------------+----------------------------------------------+
| :const:`FF` | Form feed |
+--------------+----------------------------------------------+
| :const:`CR` | Carriage return |
+--------------+----------------------------------------------+
| :const:`SO` | Shift-out, begin alternate character set |
+--------------+----------------------------------------------+
| :const:`SI` | Shift-in, resume default character set |
+--------------+----------------------------------------------+
| :const:`DLE` | Data-link escape |
+--------------+----------------------------------------------+
| :const:`DC1` | XON, for flow control |
+--------------+----------------------------------------------+
| :const:`DC2` | Device control 2, block-mode flow control |
+--------------+----------------------------------------------+
| :const:`DC3` | XOFF, for flow control |
+--------------+----------------------------------------------+
| :const:`DC4` | Device control 4 |
+--------------+----------------------------------------------+
| :const:`NAK` | Negative acknowledgement |
+--------------+----------------------------------------------+
| :const:`SYN` | Synchronous idle |
+--------------+----------------------------------------------+
| :const:`ETB` | End transmission block |
+--------------+----------------------------------------------+
| :const:`CAN` | Cancel |
+--------------+----------------------------------------------+
| :const:`EM` | End of medium |
+--------------+----------------------------------------------+
| :const:`SUB` | Substitute |
+--------------+----------------------------------------------+
| :const:`ESC` | Escape |
+--------------+----------------------------------------------+
| :const:`FS` | File separator |
+--------------+----------------------------------------------+
| :const:`GS` | Group separator |
+--------------+----------------------------------------------+
| :const:`RS` | Record separator, block-mode terminator |
+--------------+----------------------------------------------+
| :const:`US` | Unit separator |
+--------------+----------------------------------------------+
| :const:`SP` | Space |
+--------------+----------------------------------------------+
| :const:`DEL` | Delete |
+--------------+----------------------------------------------+
Note that many of these have little practical significance in modern usage. The
mnemonics derive from teleprinter conventions that predate digital computers.
The module supplies the following functions, patterned on those in the standard
C library:
.. function:: isalnum(c)
Checks for an ASCII alphanumeric character; it is equivalent to ``isalpha(c) or
isdigit(c)``.
.. function:: isalpha(c)
Checks for an ASCII alphabetic character; it is equivalent to ``isupper(c) or
islower(c)``.
.. function:: isascii(c)
Checks for a character value that fits in the 7-bit ASCII set.
.. function:: isblank(c)
Checks for an ASCII whitespace character; space or horizontal tab.
.. function:: iscntrl(c)
Checks for an ASCII control character (in the range 0x00 to 0x1f or 0x7f).
.. function:: isdigit(c)
Checks for an ASCII decimal digit, ``'0'`` through ``'9'``. This is equivalent
to ``c in string.digits``.
.. function:: isgraph(c)
Checks for ASCII any printable character except space.
.. function:: islower(c)
Checks for an ASCII lower-case character.
.. function:: isprint(c)
Checks for any ASCII printable character including space.
.. function:: ispunct(c)
Checks for any printable ASCII character which is not a space or an alphanumeric
character.
.. function:: isspace(c)
Checks for ASCII white-space characters; space, line feed, carriage return, form
feed, horizontal tab, vertical tab.
.. function:: isupper(c)
Checks for an ASCII uppercase letter.
.. function:: isxdigit(c)
Checks for an ASCII hexadecimal digit. This is equivalent to ``c in
string.hexdigits``.
.. function:: isctrl(c)
Checks for an ASCII control character (ordinal values 0 to 31).
.. function:: ismeta(c)
Checks for a non-ASCII character (ordinal values 0x80 and above).
These functions accept either integers or single-character strings; when the argument is a
string, it is first converted using the built-in function :func:`ord`.
Note that all these functions check ordinal bit values derived from the
character of the string you pass in; they do not actually know anything about
the host machine's character encoding.
The following two functions take either a single-character string or integer
byte value; they return a value of the same type.
.. function:: ascii(c)
Return the ASCII value corresponding to the low 7 bits of *c*.
.. function:: ctrl(c)
Return the control character corresponding to the given character (the character
bit value is bitwise-anded with 0x1f).
.. function:: alt(c)
Return the 8-bit character corresponding to the given ASCII character (the
character bit value is bitwise-ored with 0x80).
The following function takes either a single-character string or integer value;
it returns a string.
.. index::
single: ^ (caret); in curses module
single: ! (exclamation); in curses module
.. function:: unctrl(c)
Return a string representation of the ASCII character *c*. If *c* is printable,
this string is the character itself. If the character is a control character
(0x00--0x1f) the string consists of a caret (``'^'``) followed by the
corresponding uppercase letter. If the character is an ASCII delete (0x7f) the
string is ``'^?'``. If the character has its meta bit (0x80) set, the meta bit
is stripped, the preceding rules applied, and ``'!'`` prepended to the result.
.. data:: controlnames
A 33-element string array that contains the ASCII mnemonics for the thirty-two
ASCII control characters from 0 (NUL) to 0x1f (US), in order, plus the mnemonic
``SP`` for the space character.

120
third_party/python/Doc/library/curses.panel.rst vendored Normal file
View file

@ -0,0 +1,120 @@
:mod:`curses.panel` --- A panel stack extension for curses
==========================================================
.. module:: curses.panel
:synopsis: A panel stack extension that adds depth to curses windows.
.. sectionauthor:: A.M. Kuchling <amk@amk.ca>
--------------
Panels are windows with the added feature of depth, so they can be stacked on
top of each other, and only the visible portions of each window will be
displayed. Panels can be added, moved up or down in the stack, and removed.
.. _cursespanel-functions:
Functions
---------
The module :mod:`curses.panel` defines the following functions:
.. function:: bottom_panel()
Returns the bottom panel in the panel stack.
.. function:: new_panel(win)
Returns a panel object, associating it with the given window *win*. Be aware
that you need to keep the returned panel object referenced explicitly. If you
don't, the panel object is garbage collected and removed from the panel stack.
.. function:: top_panel()
Returns the top panel in the panel stack.
.. function:: update_panels()
Updates the virtual screen after changes in the panel stack. This does not call
:func:`curses.doupdate`, so you'll have to do this yourself.
.. _curses-panel-objects:
Panel Objects
-------------
Panel objects, as returned by :func:`new_panel` above, are windows with a
stacking order. There's always a window associated with a panel which determines
the content, while the panel methods are responsible for the window's depth in
the panel stack.
Panel objects have the following methods:
.. method:: Panel.above()
Returns the panel above the current panel.
.. method:: Panel.below()
Returns the panel below the current panel.
.. method:: Panel.bottom()
Push the panel to the bottom of the stack.
.. method:: Panel.hidden()
Returns ``True`` if the panel is hidden (not visible), ``False`` otherwise.
.. method:: Panel.hide()
Hide the panel. This does not delete the object, it just makes the window on
screen invisible.
.. method:: Panel.move(y, x)
Move the panel to the screen coordinates ``(y, x)``.
.. method:: Panel.replace(win)
Change the window associated with the panel to the window *win*.
.. method:: Panel.set_userptr(obj)
Set the panel's user pointer to *obj*. This is used to associate an arbitrary
piece of data with the panel, and can be any Python object.
.. method:: Panel.show()
Display the panel (which might have been hidden).
.. method:: Panel.top()
Push panel to the top of the stack.
.. method:: Panel.userptr()
Returns the user pointer for the panel. This might be any Python object.
.. method:: Panel.window()
Returns the window object associated with the panel.

1830
third_party/python/Doc/library/curses.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

19
third_party/python/Doc/library/custominterp.rst vendored Normal file
View file

@ -0,0 +1,19 @@
.. _custominterp:
**************************
Custom Python Interpreters
**************************
The modules described in this chapter allow writing interfaces similar to
Python's interactive interpreter. If you want a Python interpreter that
supports some special feature in addition to the Python language, you should
look at the :mod:`code` module. (The :mod:`codeop` module is lower-level, used
to support compiling a possibly-incomplete chunk of Python code.)
The full list of modules described in this chapter is:
.. toctree::
code.rst
codeop.rst

33
third_party/python/Doc/library/datatypes.rst vendored Normal file
View file

@ -0,0 +1,33 @@
.. _datatypes:
**********
Data Types
**********
The modules described in this chapter provide a variety of specialized data
types such as dates and times, fixed-type arrays, heap queues, synchronized
queues, and sets.
Python also provides some built-in data types, in particular,
:class:`dict`, :class:`list`, :class:`set` and :class:`frozenset`, and
:class:`tuple`. The :class:`str` class is used to hold
Unicode strings, and the :class:`bytes` class is used to hold binary data.
The following modules are documented in this chapter:
.. toctree::
datetime.rst
calendar.rst
collections.rst
collections.abc.rst
heapq.rst
bisect.rst
array.rst
weakref.rst
types.rst
copy.rst
pprint.rst
reprlib.rst
enum.rst

2172
third_party/python/Doc/library/datetime.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

370
third_party/python/Doc/library/dbm.rst vendored Normal file
View file

@ -0,0 +1,370 @@
:mod:`dbm` --- Interfaces to Unix "databases"
=============================================
.. module:: dbm
:synopsis: Interfaces to various Unix "database" formats.
**Source code:** :source:`Lib/dbm/__init__.py`
--------------
:mod:`dbm` is a generic interface to variants of the DBM database ---
:mod:`dbm.gnu` or :mod:`dbm.ndbm`. If none of these modules is installed, the
slow-but-simple implementation in module :mod:`dbm.dumb` will be used. There
is a `third party interface <https://www.jcea.es/programacion/pybsddb.htm>`_ to
the Oracle Berkeley DB.
.. exception:: error
A tuple containing the exceptions that can be raised by each of the supported
modules, with a unique exception also named :exc:`dbm.error` as the first
item --- the latter is used when :exc:`dbm.error` is raised.
.. function:: whichdb(filename)
This function attempts to guess which of the several simple database modules
available --- :mod:`dbm.gnu`, :mod:`dbm.ndbm` or :mod:`dbm.dumb` --- should
be used to open a given file.
Returns one of the following values: ``None`` if the file can't be opened
because it's unreadable or doesn't exist; the empty string (``''``) if the
file's format can't be guessed; or a string containing the required module
name, such as ``'dbm.ndbm'`` or ``'dbm.gnu'``.
.. function:: open(file, flag='r', mode=0o666)
Open the database file *file* and return a corresponding object.
If the database file already exists, the :func:`whichdb` function is used to
determine its type and the appropriate module is used; if it does not exist,
the first module listed above that can be imported is used.
The optional *flag* argument can be:
+---------+-------------------------------------------+
| Value | Meaning |
+=========+===========================================+
| ``'r'`` | Open existing database for reading only |
| | (default) |
+---------+-------------------------------------------+
| ``'w'`` | Open existing database for reading and |
| | writing |
+---------+-------------------------------------------+
| ``'c'`` | Open database for reading and writing, |
| | creating it if it doesn't exist |
+---------+-------------------------------------------+
| ``'n'`` | Always create a new, empty database, open |
| | for reading and writing |
+---------+-------------------------------------------+
The optional *mode* argument is the Unix mode of the file, used only when the
database has to be created. It defaults to octal ``0o666`` (and will be
modified by the prevailing umask).
The object returned by :func:`.open` supports the same basic functionality as
dictionaries; keys and their corresponding values can be stored, retrieved, and
deleted, and the :keyword:`in` operator and the :meth:`keys` method are
available, as well as :meth:`get` and :meth:`setdefault`.
.. versionchanged:: 3.2
:meth:`get` and :meth:`setdefault` are now available in all database modules.
Key and values are always stored as bytes. This means that when
strings are used they are implicitly converted to the default encoding before
being stored.
These objects also support being used in a :keyword:`with` statement, which
will automatically close them when done.
.. versionchanged:: 3.4
Added native support for the context management protocol to the objects
returned by :func:`.open`.
The following example records some hostnames and a corresponding title, and
then prints out the contents of the database::
import dbm
# Open database, creating it if necessary.
with dbm.open('cache', 'c') as db:
# Record some values
db[b'hello'] = b'there'
db['www.python.org'] = 'Python Website'
db['www.cnn.com'] = 'Cable News Network'
# Note that the keys are considered bytes now.
assert db[b'www.python.org'] == b'Python Website'
# Notice how the value is now in bytes.
assert db['www.cnn.com'] == b'Cable News Network'
# Often-used methods of the dict interface work too.
print(db.get('python.org', b'not present'))
# Storing a non-string key or value will raise an exception (most
# likely a TypeError).
db['www.yahoo.com'] = 4
# db is automatically closed when leaving the with statement.
.. seealso::
Module :mod:`shelve`
Persistence module which stores non-string data.
The individual submodules are described in the following sections.
:mod:`dbm.gnu` --- GNU's reinterpretation of dbm
------------------------------------------------
.. module:: dbm.gnu
:platform: Unix
:synopsis: GNU's reinterpretation of dbm.
**Source code:** :source:`Lib/dbm/gnu.py`
--------------
This module is quite similar to the :mod:`dbm` module, but uses the GNU library
``gdbm`` instead to provide some additional functionality. Please note that the
file formats created by :mod:`dbm.gnu` and :mod:`dbm.ndbm` are incompatible.
The :mod:`dbm.gnu` module provides an interface to the GNU DBM library.
``dbm.gnu.gdbm`` objects behave like mappings (dictionaries), except that keys and
values are always converted to bytes before storing. Printing a ``gdbm``
object doesn't print the
keys and values, and the :meth:`items` and :meth:`values` methods are not
supported.
.. exception:: error
Raised on :mod:`dbm.gnu`-specific errors, such as I/O errors. :exc:`KeyError` is
raised for general mapping errors like specifying an incorrect key.
.. function:: open(filename[, flag[, mode]])
Open a ``gdbm`` database and return a :class:`gdbm` object. The *filename*
argument is the name of the database file.
The optional *flag* argument can be:
+---------+-------------------------------------------+
| Value | Meaning |
+=========+===========================================+
| ``'r'`` | Open existing database for reading only |
| | (default) |
+---------+-------------------------------------------+
| ``'w'`` | Open existing database for reading and |
| | writing |
+---------+-------------------------------------------+
| ``'c'`` | Open database for reading and writing, |
| | creating it if it doesn't exist |
+---------+-------------------------------------------+
| ``'n'`` | Always create a new, empty database, open |
| | for reading and writing |
+---------+-------------------------------------------+
The following additional characters may be appended to the flag to control
how the database is opened:
+---------+--------------------------------------------+
| Value | Meaning |
+=========+============================================+
| ``'f'`` | Open the database in fast mode. Writes |
| | to the database will not be synchronized. |
+---------+--------------------------------------------+
| ``'s'`` | Synchronized mode. This will cause changes |
| | to the database to be immediately written |
| | to the file. |
+---------+--------------------------------------------+
| ``'u'`` | Do not lock database. |
+---------+--------------------------------------------+
Not all flags are valid for all versions of ``gdbm``. The module constant
:const:`open_flags` is a string of supported flag characters. The exception
:exc:`error` is raised if an invalid flag is specified.
The optional *mode* argument is the Unix mode of the file, used only when the
database has to be created. It defaults to octal ``0o666``.
In addition to the dictionary-like methods, ``gdbm`` objects have the
following methods:
.. method:: gdbm.firstkey()
It's possible to loop over every key in the database using this method and the
:meth:`nextkey` method. The traversal is ordered by ``gdbm``'s internal
hash values, and won't be sorted by the key values. This method returns
the starting key.
.. method:: gdbm.nextkey(key)
Returns the key that follows *key* in the traversal. The following code prints
every key in the database ``db``, without having to create a list in memory that
contains them all::
k = db.firstkey()
while k != None:
print(k)
k = db.nextkey(k)
.. method:: gdbm.reorganize()
If you have carried out a lot of deletions and would like to shrink the space
used by the ``gdbm`` file, this routine will reorganize the database. ``gdbm``
objects will not shorten the length of a database file except by using this
reorganization; otherwise, deleted file space will be kept and reused as new
(key, value) pairs are added.
.. method:: gdbm.sync()
When the database has been opened in fast mode, this method forces any
unwritten data to be written to the disk.
.. method:: gdbm.close()
Close the ``gdbm`` database.
:mod:`dbm.ndbm` --- Interface based on ndbm
-------------------------------------------
.. module:: dbm.ndbm
:platform: Unix
:synopsis: The standard "database" interface, based on ndbm.
**Source code:** :source:`Lib/dbm/ndbm.py`
--------------
The :mod:`dbm.ndbm` module provides an interface to the Unix "(n)dbm" library.
Dbm objects behave like mappings (dictionaries), except that keys and values are
always stored as bytes. Printing a ``dbm`` object doesn't print the keys and
values, and the :meth:`items` and :meth:`values` methods are not supported.
This module can be used with the "classic" ndbm interface or the GNU GDBM
compatibility interface. On Unix, the :program:`configure` script will attempt
to locate the appropriate header file to simplify building this module.
.. exception:: error
Raised on :mod:`dbm.ndbm`-specific errors, such as I/O errors. :exc:`KeyError` is raised
for general mapping errors like specifying an incorrect key.
.. data:: library
Name of the ``ndbm`` implementation library used.
.. function:: open(filename[, flag[, mode]])
Open a dbm database and return a ``ndbm`` object. The *filename* argument is the
name of the database file (without the :file:`.dir` or :file:`.pag` extensions).
The optional *flag* argument must be one of these values:
+---------+-------------------------------------------+
| Value | Meaning |
+=========+===========================================+
| ``'r'`` | Open existing database for reading only |
| | (default) |
+---------+-------------------------------------------+
| ``'w'`` | Open existing database for reading and |
| | writing |
+---------+-------------------------------------------+
| ``'c'`` | Open database for reading and writing, |
| | creating it if it doesn't exist |
+---------+-------------------------------------------+
| ``'n'`` | Always create a new, empty database, open |
| | for reading and writing |
+---------+-------------------------------------------+
The optional *mode* argument is the Unix mode of the file, used only when the
database has to be created. It defaults to octal ``0o666`` (and will be
modified by the prevailing umask).
In addition to the dictionary-like methods, ``ndbm`` objects
provide the following method:
.. method:: ndbm.close()
Close the ``ndbm`` database.
:mod:`dbm.dumb` --- Portable DBM implementation
-----------------------------------------------
.. module:: dbm.dumb
:synopsis: Portable implementation of the simple DBM interface.
**Source code:** :source:`Lib/dbm/dumb.py`
.. index:: single: databases
.. note::
The :mod:`dbm.dumb` module is intended as a last resort fallback for the
:mod:`dbm` module when a more robust module is not available. The :mod:`dbm.dumb`
module is not written for speed and is not nearly as heavily used as the other
database modules.
--------------
The :mod:`dbm.dumb` module provides a persistent dictionary-like interface which
is written entirely in Python. Unlike other modules such as :mod:`dbm.gnu` no
external library is required. As with other persistent mappings, the keys and
values are always stored as bytes.
The module defines the following:
.. exception:: error
Raised on :mod:`dbm.dumb`-specific errors, such as I/O errors. :exc:`KeyError` is
raised for general mapping errors like specifying an incorrect key.
.. function:: open(filename[, flag[, mode]])
Open a ``dumbdbm`` database and return a dumbdbm object. The *filename* argument is
the basename of the database file (without any specific extensions). When a
dumbdbm database is created, files with :file:`.dat` and :file:`.dir` extensions
are created.
The optional *flag* argument supports only the semantics of ``'c'``
and ``'n'`` values. Other values will default to database being always
opened for update, and will be created if it does not exist.
The optional *mode* argument is the Unix mode of the file, used only when the
database has to be created. It defaults to octal ``0o666`` (and will be modified
by the prevailing umask).
.. versionchanged:: 3.5
:func:`.open` always creates a new database when the flag has the value
``'n'``.
.. deprecated-removed:: 3.6 3.8
Creating database in ``'r'`` and ``'w'`` modes. Modifying database in
``'r'`` mode.
In addition to the methods provided by the
:class:`collections.abc.MutableMapping` class, :class:`dumbdbm` objects
provide the following methods:
.. method:: dumbdbm.sync()
Synchronize the on-disk directory and data files. This method is called
by the :meth:`Shelve.sync` method.
.. method:: dumbdbm.close()
Close the ``dumbdbm`` database.

18
third_party/python/Doc/library/debug.rst vendored Normal file
View file

@ -0,0 +1,18 @@
***********************
Debugging and Profiling
***********************
These libraries help you with Python development: the debugger enables you to
step through code, analyze stack frames and set breakpoints etc., and the
profilers run code and give you a detailed breakdown of execution times,
allowing you to identify bottlenecks in your programs.
.. toctree::
bdb.rst
faulthandler.rst
pdb.rst
profile.rst
timeit.rst
trace.rst
tracemalloc.rst

2112
third_party/python/Doc/library/decimal.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

26
third_party/python/Doc/library/development.rst vendored Normal file
View file

@ -0,0 +1,26 @@
.. _development:
*****************
Development Tools
*****************
The modules described in this chapter help you write software. For example, the
:mod:`pydoc` module takes a module and generates documentation based on the
module's contents. The :mod:`doctest` and :mod:`unittest` modules contains
frameworks for writing unit tests that automatically exercise code and verify
that the expected output is produced. :program:`2to3` can translate Python 2.x
source code into valid Python 3.x code.
The list of modules described in this chapter is:
.. toctree::
typing.rst
pydoc.rst
doctest.rst
unittest.rst
unittest.mock.rst
unittest.mock-examples.rst
2to3.rst
test.rst

746
third_party/python/Doc/library/difflib.rst vendored Normal file
View file

@ -0,0 +1,746 @@
:mod:`difflib` --- Helpers for computing deltas
===============================================
.. module:: difflib
:synopsis: Helpers for computing differences between objects.
.. moduleauthor:: Tim Peters <tim_one@users.sourceforge.net>
.. sectionauthor:: Tim Peters <tim_one@users.sourceforge.net>
.. Markup by Fred L. Drake, Jr. <fdrake@acm.org>
**Source code:** :source:`Lib/difflib.py`
.. testsetup::
import sys
from difflib import *
--------------
This module provides classes and functions for comparing sequences. It
can be used for example, for comparing files, and can produce difference
information in various formats, including HTML and context and unified
diffs. For comparing directories and files, see also, the :mod:`filecmp` module.
.. class:: SequenceMatcher
This is a flexible class for comparing pairs of sequences of any type, so long
as the sequence elements are :term:`hashable`. The basic algorithm predates, and is a
little fancier than, an algorithm published in the late 1980's by Ratcliff and
Obershelp under the hyperbolic name "gestalt pattern matching." The idea is to
find the longest contiguous matching subsequence that contains no "junk"
elements; these "junk" elements are ones that are uninteresting in some
sense, such as blank lines or whitespace. (Handling junk is an
extension to the Ratcliff and Obershelp algorithm.) The same
idea is then applied recursively to the pieces of the sequences to the left and
to the right of the matching subsequence. This does not yield minimal edit
sequences, but does tend to yield matches that "look right" to people.
**Timing:** The basic Ratcliff-Obershelp algorithm is cubic time in the worst
case and quadratic time in the expected case. :class:`SequenceMatcher` is
quadratic time for the worst case and has expected-case behavior dependent in a
complicated way on how many elements the sequences have in common; best case
time is linear.
**Automatic junk heuristic:** :class:`SequenceMatcher` supports a heuristic that
automatically treats certain sequence items as junk. The heuristic counts how many
times each individual item appears in the sequence. If an item's duplicates (after
the first one) account for more than 1% of the sequence and the sequence is at least
200 items long, this item is marked as "popular" and is treated as junk for
the purpose of sequence matching. This heuristic can be turned off by setting
the ``autojunk`` argument to ``False`` when creating the :class:`SequenceMatcher`.
.. versionadded:: 3.2
The *autojunk* parameter.
.. class:: Differ
This is a class for comparing sequences of lines of text, and producing
human-readable differences or deltas. Differ uses :class:`SequenceMatcher`
both to compare sequences of lines, and to compare sequences of characters
within similar (near-matching) lines.
Each line of a :class:`Differ` delta begins with a two-letter code:
+----------+-------------------------------------------+
| Code | Meaning |
+==========+===========================================+
| ``'- '`` | line unique to sequence 1 |
+----------+-------------------------------------------+
| ``'+ '`` | line unique to sequence 2 |
+----------+-------------------------------------------+
| ``' '`` | line common to both sequences |
+----------+-------------------------------------------+
| ``'? '`` | line not present in either input sequence |
+----------+-------------------------------------------+
Lines beginning with '``?``' attempt to guide the eye to intraline differences,
and were not present in either input sequence. These lines can be confusing if
the sequences contain tab characters.
.. class:: HtmlDiff
This class can be used to create an HTML table (or a complete HTML file
containing the table) showing a side by side, line by line comparison of text
with inter-line and intra-line change highlights. The table can be generated in
either full or contextual difference mode.
The constructor for this class is:
.. method:: __init__(tabsize=8, wrapcolumn=None, linejunk=None, charjunk=IS_CHARACTER_JUNK)
Initializes instance of :class:`HtmlDiff`.
*tabsize* is an optional keyword argument to specify tab stop spacing and
defaults to ``8``.
*wrapcolumn* is an optional keyword to specify column number where lines are
broken and wrapped, defaults to ``None`` where lines are not wrapped.
*linejunk* and *charjunk* are optional keyword arguments passed into :func:`ndiff`
(used by :class:`HtmlDiff` to generate the side by side HTML differences). See
:func:`ndiff` documentation for argument default values and descriptions.
The following methods are public:
.. method:: make_file(fromlines, tolines, fromdesc='', todesc='', context=False, \
numlines=5, *, charset='utf-8')
Compares *fromlines* and *tolines* (lists of strings) and returns a string which
is a complete HTML file containing a table showing line by line differences with
inter-line and intra-line changes highlighted.
*fromdesc* and *todesc* are optional keyword arguments to specify from/to file
column header strings (both default to an empty string).
*context* and *numlines* are both optional keyword arguments. Set *context* to
``True`` when contextual differences are to be shown, else the default is
``False`` to show the full files. *numlines* defaults to ``5``. When *context*
is ``True`` *numlines* controls the number of context lines which surround the
difference highlights. When *context* is ``False`` *numlines* controls the
number of lines which are shown before a difference highlight when using the
"next" hyperlinks (setting to zero would cause the "next" hyperlinks to place
the next difference highlight at the top of the browser without any leading
context).
.. versionchanged:: 3.5
*charset* keyword-only argument was added. The default charset of
HTML document changed from ``'ISO-8859-1'`` to ``'utf-8'``.
.. method:: make_table(fromlines, tolines, fromdesc='', todesc='', context=False, numlines=5)
Compares *fromlines* and *tolines* (lists of strings) and returns a string which
is a complete HTML table showing line by line differences with inter-line and
intra-line changes highlighted.
The arguments for this method are the same as those for the :meth:`make_file`
method.
:file:`Tools/scripts/diff.py` is a command-line front-end to this class and
contains a good example of its use.
.. function:: context_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\\n')
Compare *a* and *b* (lists of strings); return a delta (a :term:`generator`
generating the delta lines) in context diff format.
Context diffs are a compact way of showing just the lines that have changed plus
a few lines of context. The changes are shown in a before/after style. The
number of context lines is set by *n* which defaults to three.
By default, the diff control lines (those with ``***`` or ``---``) are created
with a trailing newline. This is helpful so that inputs created from
:func:`io.IOBase.readlines` result in diffs that are suitable for use with
:func:`io.IOBase.writelines` since both the inputs and outputs have trailing
newlines.
For inputs that do not have trailing newlines, set the *lineterm* argument to
``""`` so that the output will be uniformly newline free.
The context diff format normally has a header for filenames and modification
times. Any or all of these may be specified using strings for *fromfile*,
*tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally
expressed in the ISO 8601 format. If not specified, the
strings default to blanks.
>>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n']
>>> s2 = ['python\n', 'eggy\n', 'hamster\n', 'guido\n']
>>> sys.stdout.writelines(context_diff(s1, s2, fromfile='before.py', tofile='after.py'))
*** before.py
--- after.py
***************
*** 1,4 ****
! bacon
! eggs
! ham
guido
--- 1,4 ----
! python
! eggy
! hamster
guido
See :ref:`difflib-interface` for a more detailed example.
.. function:: get_close_matches(word, possibilities, n=3, cutoff=0.6)
Return a list of the best "good enough" matches. *word* is a sequence for which
close matches are desired (typically a string), and *possibilities* is a list of
sequences against which to match *word* (typically a list of strings).
Optional argument *n* (default ``3``) is the maximum number of close matches to
return; *n* must be greater than ``0``.
Optional argument *cutoff* (default ``0.6``) is a float in the range [0, 1].
Possibilities that don't score at least that similar to *word* are ignored.
The best (no more than *n*) matches among the possibilities are returned in a
list, sorted by similarity score, most similar first.
>>> get_close_matches('appel', ['ape', 'apple', 'peach', 'puppy'])
['apple', 'ape']
>>> import keyword
>>> get_close_matches('wheel', keyword.kwlist)
['while']
>>> get_close_matches('pineapple', keyword.kwlist)
[]
>>> get_close_matches('accept', keyword.kwlist)
['except']
.. function:: ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK)
Compare *a* and *b* (lists of strings); return a :class:`Differ`\ -style
delta (a :term:`generator` generating the delta lines).
Optional keyword parameters *linejunk* and *charjunk* are filtering functions
(or ``None``):
*linejunk*: A function that accepts a single string argument, and returns
true if the string is junk, or false if not. The default is ``None``. There
is also a module-level function :func:`IS_LINE_JUNK`, which filters out lines
without visible characters, except for at most one pound character (``'#'``)
-- however the underlying :class:`SequenceMatcher` class does a dynamic
analysis of which lines are so frequent as to constitute noise, and this
usually works better than using this function.
*charjunk*: A function that accepts a character (a string of length 1), and
returns if the character is junk, or false if not. The default is module-level
function :func:`IS_CHARACTER_JUNK`, which filters out whitespace characters (a
blank or tab; it's a bad idea to include newline in this!).
:file:`Tools/scripts/ndiff.py` is a command-line front-end to this function.
>>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True),
... 'ore\ntree\nemu\n'.splitlines(keepends=True))
>>> print(''.join(diff), end="")
- one
? ^
+ ore
? ^
- two
- three
? -
+ tree
+ emu
.. function:: restore(sequence, which)
Return one of the two sequences that generated a delta.
Given a *sequence* produced by :meth:`Differ.compare` or :func:`ndiff`, extract
lines originating from file 1 or 2 (parameter *which*), stripping off line
prefixes.
Example:
>>> diff = ndiff('one\ntwo\nthree\n'.splitlines(keepends=True),
... 'ore\ntree\nemu\n'.splitlines(keepends=True))
>>> diff = list(diff) # materialize the generated delta into a list
>>> print(''.join(restore(diff, 1)), end="")
one
two
three
>>> print(''.join(restore(diff, 2)), end="")
ore
tree
emu
.. function:: unified_diff(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\\n')
Compare *a* and *b* (lists of strings); return a delta (a :term:`generator`
generating the delta lines) in unified diff format.
Unified diffs are a compact way of showing just the lines that have changed plus
a few lines of context. The changes are shown in an inline style (instead of
separate before/after blocks). The number of context lines is set by *n* which
defaults to three.
By default, the diff control lines (those with ``---``, ``+++``, or ``@@``) are
created with a trailing newline. This is helpful so that inputs created from
:func:`io.IOBase.readlines` result in diffs that are suitable for use with
:func:`io.IOBase.writelines` since both the inputs and outputs have trailing
newlines.
For inputs that do not have trailing newlines, set the *lineterm* argument to
``""`` so that the output will be uniformly newline free.
The context diff format normally has a header for filenames and modification
times. Any or all of these may be specified using strings for *fromfile*,
*tofile*, *fromfiledate*, and *tofiledate*. The modification times are normally
expressed in the ISO 8601 format. If not specified, the
strings default to blanks.
>>> s1 = ['bacon\n', 'eggs\n', 'ham\n', 'guido\n']
>>> s2 = ['python\n', 'eggy\n', 'hamster\n', 'guido\n']
>>> sys.stdout.writelines(unified_diff(s1, s2, fromfile='before.py', tofile='after.py'))
--- before.py
+++ after.py
@@ -1,4 +1,4 @@
-bacon
-eggs
-ham
+python
+eggy
+hamster
guido
See :ref:`difflib-interface` for a more detailed example.
.. function:: diff_bytes(dfunc, a, b, fromfile=b'', tofile=b'', fromfiledate=b'', tofiledate=b'', n=3, lineterm=b'\\n')
Compare *a* and *b* (lists of bytes objects) using *dfunc*; yield a
sequence of delta lines (also bytes) in the format returned by *dfunc*.
*dfunc* must be a callable, typically either :func:`unified_diff` or
:func:`context_diff`.
Allows you to compare data with unknown or inconsistent encoding. All
inputs except *n* must be bytes objects, not str. Works by losslessly
converting all inputs (except *n*) to str, and calling ``dfunc(a, b,
fromfile, tofile, fromfiledate, tofiledate, n, lineterm)``. The output of
*dfunc* is then converted back to bytes, so the delta lines that you
receive have the same unknown/inconsistent encodings as *a* and *b*.
.. versionadded:: 3.5
.. function:: IS_LINE_JUNK(line)
Return true for ignorable lines. The line *line* is ignorable if *line* is
blank or contains a single ``'#'``, otherwise it is not ignorable. Used as a
default for parameter *linejunk* in :func:`ndiff` in older versions.
.. function:: IS_CHARACTER_JUNK(ch)
Return true for ignorable characters. The character *ch* is ignorable if *ch*
is a space or tab, otherwise it is not ignorable. Used as a default for
parameter *charjunk* in :func:`ndiff`.
.. seealso::
`Pattern Matching: The Gestalt Approach <http://www.drdobbs.com/database/pattern-matching-the-gestalt-approach/184407970>`_
Discussion of a similar algorithm by John W. Ratcliff and D. E. Metzener. This
was published in `Dr. Dobb's Journal <http://www.drdobbs.com/>`_ in July, 1988.
.. _sequence-matcher:
SequenceMatcher Objects
-----------------------
The :class:`SequenceMatcher` class has this constructor:
.. class:: SequenceMatcher(isjunk=None, a='', b='', autojunk=True)
Optional argument *isjunk* must be ``None`` (the default) or a one-argument
function that takes a sequence element and returns true if and only if the
element is "junk" and should be ignored. Passing ``None`` for *isjunk* is
equivalent to passing ``lambda x: 0``; in other words, no elements are ignored.
For example, pass::
lambda x: x in " \t"
if you're comparing lines as sequences of characters, and don't want to synch up
on blanks or hard tabs.
The optional arguments *a* and *b* are sequences to be compared; both default to
empty strings. The elements of both sequences must be :term:`hashable`.
The optional argument *autojunk* can be used to disable the automatic junk
heuristic.
.. versionadded:: 3.2
The *autojunk* parameter.
SequenceMatcher objects get three data attributes: *bjunk* is the
set of elements of *b* for which *isjunk* is ``True``; *bpopular* is the set of
non-junk elements considered popular by the heuristic (if it is not
disabled); *b2j* is a dict mapping the remaining elements of *b* to a list
of positions where they occur. All three are reset whenever *b* is reset
with :meth:`set_seqs` or :meth:`set_seq2`.
.. versionadded:: 3.2
The *bjunk* and *bpopular* attributes.
:class:`SequenceMatcher` objects have the following methods:
.. method:: set_seqs(a, b)
Set the two sequences to be compared.
:class:`SequenceMatcher` computes and caches detailed information about the
second sequence, so if you want to compare one sequence against many
sequences, use :meth:`set_seq2` to set the commonly used sequence once and
call :meth:`set_seq1` repeatedly, once for each of the other sequences.
.. method:: set_seq1(a)
Set the first sequence to be compared. The second sequence to be compared
is not changed.
.. method:: set_seq2(b)
Set the second sequence to be compared. The first sequence to be compared
is not changed.
.. method:: find_longest_match(alo, ahi, blo, bhi)
Find longest matching block in ``a[alo:ahi]`` and ``b[blo:bhi]``.
If *isjunk* was omitted or ``None``, :meth:`find_longest_match` returns
``(i, j, k)`` such that ``a[i:i+k]`` is equal to ``b[j:j+k]``, where ``alo
<= i <= i+k <= ahi`` and ``blo <= j <= j+k <= bhi``. For all ``(i', j',
k')`` meeting those conditions, the additional conditions ``k >= k'``, ``i
<= i'``, and if ``i == i'``, ``j <= j'`` are also met. In other words, of
all maximal matching blocks, return one that starts earliest in *a*, and
of all those maximal matching blocks that start earliest in *a*, return
the one that starts earliest in *b*.
>>> s = SequenceMatcher(None, " abcd", "abcd abcd")
>>> s.find_longest_match(0, 5, 0, 9)
Match(a=0, b=4, size=5)
If *isjunk* was provided, first the longest matching block is determined
as above, but with the additional restriction that no junk element appears
in the block. Then that block is extended as far as possible by matching
(only) junk elements on both sides. So the resulting block never matches
on junk except as identical junk happens to be adjacent to an interesting
match.
Here's the same example as before, but considering blanks to be junk. That
prevents ``' abcd'`` from matching the ``' abcd'`` at the tail end of the
second sequence directly. Instead only the ``'abcd'`` can match, and
matches the leftmost ``'abcd'`` in the second sequence:
>>> s = SequenceMatcher(lambda x: x==" ", " abcd", "abcd abcd")
>>> s.find_longest_match(0, 5, 0, 9)
Match(a=1, b=0, size=4)
If no blocks match, this returns ``(alo, blo, 0)``.
This method returns a :term:`named tuple` ``Match(a, b, size)``.
.. method:: get_matching_blocks()
Return list of triples describing non-overlapping matching subsequences.
Each triple is of the form ``(i, j, n)``,
and means that ``a[i:i+n] == b[j:j+n]``. The
triples are monotonically increasing in *i* and *j*.
The last triple is a dummy, and has the value ``(len(a), len(b), 0)``. It
is the only triple with ``n == 0``. If ``(i, j, n)`` and ``(i', j', n')``
are adjacent triples in the list, and the second is not the last triple in
the list, then ``i+n < i'`` or ``j+n < j'``; in other words, adjacent
triples always describe non-adjacent equal blocks.
.. XXX Explain why a dummy is used!
.. doctest::
>>> s = SequenceMatcher(None, "abxcd", "abcd")
>>> s.get_matching_blocks()
[Match(a=0, b=0, size=2), Match(a=3, b=2, size=2), Match(a=5, b=4, size=0)]
.. method:: get_opcodes()
Return list of 5-tuples describing how to turn *a* into *b*. Each tuple is
of the form ``(tag, i1, i2, j1, j2)``. The first tuple has ``i1 == j1 ==
0``, and remaining tuples have *i1* equal to the *i2* from the preceding
tuple, and, likewise, *j1* equal to the previous *j2*.
The *tag* values are strings, with these meanings:
+---------------+---------------------------------------------+
| Value | Meaning |
+===============+=============================================+
| ``'replace'`` | ``a[i1:i2]`` should be replaced by |
| | ``b[j1:j2]``. |
+---------------+---------------------------------------------+
| ``'delete'`` | ``a[i1:i2]`` should be deleted. Note that |
| | ``j1 == j2`` in this case. |
+---------------+---------------------------------------------+
| ``'insert'`` | ``b[j1:j2]`` should be inserted at |
| | ``a[i1:i1]``. Note that ``i1 == i2`` in |
| | this case. |
+---------------+---------------------------------------------+
| ``'equal'`` | ``a[i1:i2] == b[j1:j2]`` (the sub-sequences |
| | are equal). |
+---------------+---------------------------------------------+
For example::
>>> a = "qabxcd"
>>> b = "abycdf"
>>> s = SequenceMatcher(None, a, b)
>>> for tag, i1, i2, j1, j2 in s.get_opcodes():
... print('{:7} a[{}:{}] --> b[{}:{}] {!r:>8} --> {!r}'.format(
... tag, i1, i2, j1, j2, a[i1:i2], b[j1:j2]))
delete a[0:1] --> b[0:0] 'q' --> ''
equal a[1:3] --> b[0:2] 'ab' --> 'ab'
replace a[3:4] --> b[2:3] 'x' --> 'y'
equal a[4:6] --> b[3:5] 'cd' --> 'cd'
insert a[6:6] --> b[5:6] '' --> 'f'
.. method:: get_grouped_opcodes(n=3)
Return a :term:`generator` of groups with up to *n* lines of context.
Starting with the groups returned by :meth:`get_opcodes`, this method
splits out smaller change clusters and eliminates intervening ranges which
have no changes.
The groups are returned in the same format as :meth:`get_opcodes`.
.. method:: ratio()
Return a measure of the sequences' similarity as a float in the range [0,
1].
Where T is the total number of elements in both sequences, and M is the
number of matches, this is 2.0\*M / T. Note that this is ``1.0`` if the
sequences are identical, and ``0.0`` if they have nothing in common.
This is expensive to compute if :meth:`get_matching_blocks` or
:meth:`get_opcodes` hasn't already been called, in which case you may want
to try :meth:`quick_ratio` or :meth:`real_quick_ratio` first to get an
upper bound.
.. method:: quick_ratio()
Return an upper bound on :meth:`ratio` relatively quickly.
.. method:: real_quick_ratio()
Return an upper bound on :meth:`ratio` very quickly.
The three methods that return the ratio of matching to total characters can give
different results due to differing levels of approximation, although
:meth:`quick_ratio` and :meth:`real_quick_ratio` are always at least as large as
:meth:`ratio`:
>>> s = SequenceMatcher(None, "abcd", "bcde")
>>> s.ratio()
0.75
>>> s.quick_ratio()
0.75
>>> s.real_quick_ratio()
1.0
.. _sequencematcher-examples:
SequenceMatcher Examples
------------------------
This example compares two strings, considering blanks to be "junk":
>>> s = SequenceMatcher(lambda x: x == " ",
... "private Thread currentThread;",
... "private volatile Thread currentThread;")
:meth:`ratio` returns a float in [0, 1], measuring the similarity of the
sequences. As a rule of thumb, a :meth:`ratio` value over 0.6 means the
sequences are close matches:
>>> print(round(s.ratio(), 3))
0.866
If you're only interested in where the sequences match,
:meth:`get_matching_blocks` is handy:
>>> for block in s.get_matching_blocks():
... print("a[%d] and b[%d] match for %d elements" % block)
a[0] and b[0] match for 8 elements
a[8] and b[17] match for 21 elements
a[29] and b[38] match for 0 elements
Note that the last tuple returned by :meth:`get_matching_blocks` is always a
dummy, ``(len(a), len(b), 0)``, and this is the only case in which the last
tuple element (number of elements matched) is ``0``.
If you want to know how to change the first sequence into the second, use
:meth:`get_opcodes`:
>>> for opcode in s.get_opcodes():
... print("%6s a[%d:%d] b[%d:%d]" % opcode)
equal a[0:8] b[0:8]
insert a[8:8] b[8:17]
equal a[8:29] b[17:38]
.. seealso::
* The :func:`get_close_matches` function in this module which shows how
simple code building on :class:`SequenceMatcher` can be used to do useful
work.
* `Simple version control recipe
<https://code.activestate.com/recipes/576729/>`_ for a small application
built with :class:`SequenceMatcher`.
.. _differ-objects:
Differ Objects
--------------
Note that :class:`Differ`\ -generated deltas make no claim to be **minimal**
diffs. To the contrary, minimal diffs are often counter-intuitive, because they
synch up anywhere possible, sometimes accidental matches 100 pages apart.
Restricting synch points to contiguous matches preserves some notion of
locality, at the occasional cost of producing a longer diff.
The :class:`Differ` class has this constructor:
.. class:: Differ(linejunk=None, charjunk=None)
Optional keyword parameters *linejunk* and *charjunk* are for filter functions
(or ``None``):
*linejunk*: A function that accepts a single string argument, and returns true
if the string is junk. The default is ``None``, meaning that no line is
considered junk.
*charjunk*: A function that accepts a single character argument (a string of
length 1), and returns true if the character is junk. The default is ``None``,
meaning that no character is considered junk.
These junk-filtering functions speed up matching to find
differences and do not cause any differing lines or characters to
be ignored. Read the description of the
:meth:`~SequenceMatcher.find_longest_match` method's *isjunk*
parameter for an explanation.
:class:`Differ` objects are used (deltas generated) via a single method:
.. method:: Differ.compare(a, b)
Compare two sequences of lines, and generate the delta (a sequence of lines).
Each sequence must contain individual single-line strings ending with
newlines. Such sequences can be obtained from the
:meth:`~io.IOBase.readlines` method of file-like objects. The delta
generated also consists of newline-terminated strings, ready to be
printed as-is via the :meth:`~io.IOBase.writelines` method of a
file-like object.
.. _differ-examples:
Differ Example
--------------
This example compares two texts. First we set up the texts, sequences of
individual single-line strings ending with newlines (such sequences can also be
obtained from the :meth:`~io.BaseIO.readlines` method of file-like objects):
>>> text1 = ''' 1. Beautiful is better than ugly.
... 2. Explicit is better than implicit.
... 3. Simple is better than complex.
... 4. Complex is better than complicated.
... '''.splitlines(keepends=True)
>>> len(text1)
4
>>> text1[0][-1]
'\n'
>>> text2 = ''' 1. Beautiful is better than ugly.
... 3. Simple is better than complex.
... 4. Complicated is better than complex.
... 5. Flat is better than nested.
... '''.splitlines(keepends=True)
Next we instantiate a Differ object:
>>> d = Differ()
Note that when instantiating a :class:`Differ` object we may pass functions to
filter out line and character "junk." See the :meth:`Differ` constructor for
details.
Finally, we compare the two:
>>> result = list(d.compare(text1, text2))
``result`` is a list of strings, so let's pretty-print it:
>>> from pprint import pprint
>>> pprint(result)
[' 1. Beautiful is better than ugly.\n',
'- 2. Explicit is better than implicit.\n',
'- 3. Simple is better than complex.\n',
'+ 3. Simple is better than complex.\n',
'? ++\n',
'- 4. Complex is better than complicated.\n',
'? ^ ---- ^\n',
'+ 4. Complicated is better than complex.\n',
'? ++++ ^ ^\n',
'+ 5. Flat is better than nested.\n']
As a single multi-line string it looks like this:
>>> import sys
>>> sys.stdout.writelines(result)
1. Beautiful is better than ugly.
- 2. Explicit is better than implicit.
- 3. Simple is better than complex.
+ 3. Simple is better than complex.
? ++
- 4. Complex is better than complicated.
? ^ ---- ^
+ 4. Complicated is better than complex.
? ++++ ^ ^
+ 5. Flat is better than nested.
.. _difflib-interface:
A command-line interface to difflib
-----------------------------------
This example shows how to use difflib to create a ``diff``-like utility.
It is also contained in the Python source distribution, as
:file:`Tools/scripts/diff.py`.
.. literalinclude:: ../../Tools/scripts/diff.py

1223
third_party/python/Doc/library/dis.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

15
third_party/python/Doc/library/distribution.rst vendored Normal file
View file

@ -0,0 +1,15 @@
***********************************
Software Packaging and Distribution
***********************************
These libraries help you with publishing and installing Python software.
While these modules are designed to work in conjunction with the
`Python Package Index <https://pypi.org>`__, they can also be used
with a local index server, or without any index server at all.
.. toctree::
distutils.rst
ensurepip.rst
venv.rst
zipapp.rst

44
third_party/python/Doc/library/distutils.rst vendored Normal file
View file

@ -0,0 +1,44 @@
:mod:`distutils` --- Building and installing Python modules
===========================================================
.. module:: distutils
:synopsis: Support for building and installing Python modules into an
existing Python installation.
.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
--------------
The :mod:`distutils` package provides support for building and installing
additional modules into a Python installation. The new modules may be either
100%-pure Python, or may be extension modules written in C, or may be
collections of Python packages which include modules coded in both Python and C.
Most Python users will *not* want to use this module directly, but instead
use the cross-version tools maintained by the Python Packaging Authority. In
particular,
`setuptools <https://setuptools.readthedocs.io/en/latest/>`__ is an
enhanced alternative to :mod:`distutils` that provides:
* support for declaring project dependencies
* additional mechanisms for configuring which files to include in source
releases (including plugins for integration with version control systems)
* the ability to declare project "entry points", which can be used as the
basis for application plugin systems
* the ability to automatically generate Windows command line executables at
installation time rather than needing to prebuild them
* consistent behaviour across all supported Python versions
The recommended `pip <https://pip.pypa.io/>`__ installer runs all
``setup.py`` scripts with ``setuptools``, even if the script itself only
imports ``distutils``. Refer to the
`Python Packaging User Guide <https://packaging.python.org>`_ for more
information.
For the benefits of packaging tool authors and users seeking a deeper
understanding of the details of the current packaging and distribution
system, the legacy :mod:`distutils` based user documentation and API
reference remain available:
* :ref:`install-index`
* :ref:`distutils-index`

1864
third_party/python/Doc/library/doctest.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

25
third_party/python/Doc/library/dummy_threading.rst vendored Normal file
View file

@ -0,0 +1,25 @@
:mod:`dummy_threading` --- Drop-in replacement for the :mod:`threading` module
==============================================================================
.. module:: dummy_threading
:synopsis: Drop-in replacement for the threading module.
**Source code:** :source:`Lib/dummy_threading.py`
--------------
This module provides a duplicate interface to the :mod:`threading` module. It
is meant to be imported when the :mod:`_thread` module is not provided on a
platform.
Suggested usage is::
try:
import threading
except ImportError:
import dummy_threading as threading
Be careful to not use this module where deadlock might occur from a thread being
created that blocks waiting for another thread to be created. This often occurs
with blocking I/O.

247
third_party/python/Doc/library/email.charset.rst vendored Normal file
View file

@ -0,0 +1,247 @@
:mod:`email.charset`: Representing character sets
-------------------------------------------------
.. module:: email.charset
:synopsis: Character Sets
**Source code:** :source:`Lib/email/charset.py`
--------------
This module is part of the legacy (``Compat32``) email API. In the new
API only the aliases table is used.
The remaining text in this section is the original documentation of the module.
This module provides a class :class:`Charset` for representing character sets
and character set conversions in email messages, as well as a character set
registry and several convenience methods for manipulating this registry.
Instances of :class:`Charset` are used in several other modules within the
:mod:`email` package.
Import this class from the :mod:`email.charset` module.
.. class:: Charset(input_charset=DEFAULT_CHARSET)
Map character sets to their email properties.
This class provides information about the requirements imposed on email for a
specific character set. It also provides convenience routines for converting
between character sets, given the availability of the applicable codecs. Given
a character set, it will do its best to provide information on how to use that
character set in an email message in an RFC-compliant way.
Certain character sets must be encoded with quoted-printable or base64 when used
in email headers or bodies. Certain character sets must be converted outright,
and are not allowed in email.
Optional *input_charset* is as described below; it is always coerced to lower
case. After being alias normalized it is also used as a lookup into the
registry of character sets to find out the header encoding, body encoding, and
output conversion codec to be used for the character set. For example, if
*input_charset* is ``iso-8859-1``, then headers and bodies will be encoded using
quoted-printable and no output conversion codec is necessary. If
*input_charset* is ``euc-jp``, then headers will be encoded with base64, bodies
will not be encoded, but output text will be converted from the ``euc-jp``
character set to the ``iso-2022-jp`` character set.
:class:`Charset` instances have the following data attributes:
.. attribute:: input_charset
The initial character set specified. Common aliases are converted to
their *official* email names (e.g. ``latin_1`` is converted to
``iso-8859-1``). Defaults to 7-bit ``us-ascii``.
.. attribute:: header_encoding
If the character set must be encoded before it can be used in an email
header, this attribute will be set to ``Charset.QP`` (for
quoted-printable), ``Charset.BASE64`` (for base64 encoding), or
``Charset.SHORTEST`` for the shortest of QP or BASE64 encoding. Otherwise,
it will be ``None``.
.. attribute:: body_encoding
Same as *header_encoding*, but describes the encoding for the mail
message's body, which indeed may be different than the header encoding.
``Charset.SHORTEST`` is not allowed for *body_encoding*.
.. attribute:: output_charset
Some character sets must be converted before they can be used in email
headers or bodies. If the *input_charset* is one of them, this attribute
will contain the name of the character set output will be converted to.
Otherwise, it will be ``None``.
.. attribute:: input_codec
The name of the Python codec used to convert the *input_charset* to
Unicode. If no conversion codec is necessary, this attribute will be
``None``.
.. attribute:: output_codec
The name of the Python codec used to convert Unicode to the
*output_charset*. If no conversion codec is necessary, this attribute
will have the same value as the *input_codec*.
:class:`Charset` instances also have the following methods:
.. method:: get_body_encoding()
Return the content transfer encoding used for body encoding.
This is either the string ``quoted-printable`` or ``base64`` depending on
the encoding used, or it is a function, in which case you should call the
function with a single argument, the Message object being encoded. The
function should then set the :mailheader:`Content-Transfer-Encoding`
header itself to whatever is appropriate.
Returns the string ``quoted-printable`` if *body_encoding* is ``QP``,
returns the string ``base64`` if *body_encoding* is ``BASE64``, and
returns the string ``7bit`` otherwise.
.. XXX to_splittable and from_splittable are not there anymore!
.. to_splittable(s)
Convert a possibly multibyte string to a safely splittable format. *s* is
the string to split.
Uses the *input_codec* to try and convert the string to Unicode, so it can
be safely split on character boundaries (even for multibyte characters).
Returns the string as-is if it isn't known how to convert *s* to Unicode
with the *input_charset*.
Characters that could not be converted to Unicode will be replaced with
the Unicode replacement character ``'U+FFFD'``.
.. from_splittable(ustr[, to_output])
Convert a splittable string back into an encoded string. *ustr* is a
Unicode string to "unsplit".
This method uses the proper codec to try and convert the string from
Unicode back into an encoded format. Return the string as-is if it is not
Unicode, or if it could not be converted from Unicode.
Characters that could not be converted from Unicode will be replaced with
an appropriate character (usually ``'?'``).
If *to_output* is ``True`` (the default), uses *output_codec* to convert
to an encoded format. If *to_output* is ``False``, it uses *input_codec*.
.. method:: get_output_charset()
Return the output character set.
This is the *output_charset* attribute if that is not ``None``, otherwise
it is *input_charset*.
.. method:: header_encode(string)
Header-encode the string *string*.
The type of encoding (base64 or quoted-printable) will be based on the
*header_encoding* attribute.
.. method:: header_encode_lines(string, maxlengths)
Header-encode a *string* by converting it first to bytes.
This is similar to :meth:`header_encode` except that the string is fit
into maximum line lengths as given by the argument *maxlengths*, which
must be an iterator: each element returned from this iterator will provide
the next maximum line length.
.. method:: body_encode(string)
Body-encode the string *string*.
The type of encoding (base64 or quoted-printable) will be based on the
*body_encoding* attribute.
The :class:`Charset` class also provides a number of methods to support
standard operations and built-in functions.
.. method:: __str__()
Returns *input_charset* as a string coerced to lower
case. :meth:`__repr__` is an alias for :meth:`__str__`.
.. method:: __eq__(other)
This method allows you to compare two :class:`Charset` instances for
equality.
.. method:: __ne__(other)
This method allows you to compare two :class:`Charset` instances for
inequality.
The :mod:`email.charset` module also provides the following functions for adding
new entries to the global character set, alias, and codec registries:
.. function:: add_charset(charset, header_enc=None, body_enc=None, output_charset=None)
Add character properties to the global registry.
*charset* is the input character set, and must be the canonical name of a
character set.
Optional *header_enc* and *body_enc* is either ``Charset.QP`` for
quoted-printable, ``Charset.BASE64`` for base64 encoding,
``Charset.SHORTEST`` for the shortest of quoted-printable or base64 encoding,
or ``None`` for no encoding. ``SHORTEST`` is only valid for
*header_enc*. The default is ``None`` for no encoding.
Optional *output_charset* is the character set that the output should be in.
Conversions will proceed from input charset, to Unicode, to the output charset
when the method :meth:`Charset.convert` is called. The default is to output in
the same character set as the input.
Both *input_charset* and *output_charset* must have Unicode codec entries in the
module's character set-to-codec mapping; use :func:`add_codec` to add codecs the
module does not know about. See the :mod:`codecs` module's documentation for
more information.
The global character set registry is kept in the module global dictionary
``CHARSETS``.
.. function:: add_alias(alias, canonical)
Add a character set alias. *alias* is the alias name, e.g. ``latin-1``.
*canonical* is the character set's canonical name, e.g. ``iso-8859-1``.
The global charset alias registry is kept in the module global dictionary
``ALIASES``.
.. function:: add_codec(charset, codecname)
Add a codec that map characters in the given character set to and from Unicode.
*charset* is the canonical name of a character set. *codecname* is the name of a
Python codec, as appropriate for the second argument to the :class:`str`'s
:meth:`~str.encode` method.

755
third_party/python/Doc/library/email.compat32-message.rst vendored Normal file
View file

@ -0,0 +1,755 @@
.. _compat32_message:
:mod:`email.message.Message`: Representing an email message using the :data:`~email.policy.compat32` API
--------------------------------------------------------------------------------------------------------
.. module:: email.message
:synopsis: The base class representing email messages in a fashion
backward compatible with python3.2
:noindex:
The :class:`Message` class is very similar to the
:class:`~email.message.EmailMessage` class, without the methods added by that
class, and with the default behavior of certain other methods being slightly
different. We also document here some methods that, while supported by the
:class:`~email.message.EmailMessage` class, are not recommended unless you are
dealing with legacy code.
The philosophy and structure of the two classes is otherwise the same.
This document describes the behavior under the default (for :class:`Message`)
policy :attr:`~email.policy.Compat32`. If you are going to use another policy,
you should be using the :class:`~email.message.EmailMessage` class instead.
An email message consists of *headers* and a *payload*. Headers must be
:rfc:`5233` style names and values, where the field name and value are
separated by a colon. The colon is not part of either the field name or the
field value. The payload may be a simple text message, or a binary object, or
a structured sequence of sub-messages each with their own set of headers and
their own payload. The latter type of payload is indicated by the message
having a MIME type such as :mimetype:`multipart/\*` or
:mimetype:`message/rfc822`.
The conceptual model provided by a :class:`Message` object is that of an
ordered dictionary of headers with additional methods for accessing both
specialized information from the headers, for accessing the payload, for
generating a serialized version of the message, and for recursively walking
over the object tree. Note that duplicate headers are supported but special
methods must be used to access them.
The :class:`Message` pseudo-dictionary is indexed by the header names, which
must be ASCII values. The values of the dictionary are strings that are
supposed to contain only ASCII characters; there is some special handling for
non-ASCII input, but it doesn't always produce the correct results. Headers
are stored and returned in case-preserving form, but field names are matched
case-insensitively. There may also be a single envelope header, also known as
the *Unix-From* header or the ``From_`` header. The *payload* is either a
string or bytes, in the case of simple message objects, or a list of
:class:`Message` objects, for MIME container documents (e.g.
:mimetype:`multipart/\*` and :mimetype:`message/rfc822`).
Here are the methods of the :class:`Message` class:
.. class:: Message(policy=compat32)
If *policy* is specified (it must be an instance of a :mod:`~email.policy`
class) use the rules it specifies to update and serialize the representation
of the message. If *policy* is not set, use the :class:`compat32
<email.policy.Compat32>` policy, which maintains backward compatibility with
the Python 3.2 version of the email package. For more information see the
:mod:`~email.policy` documentation.
.. versionchanged:: 3.3 The *policy* keyword argument was added.
.. method:: as_string(unixfrom=False, maxheaderlen=0, policy=None)
Return the entire message flattened as a string. When optional *unixfrom*
is true, the envelope header is included in the returned string.
*unixfrom* defaults to ``False``. For backward compatibility reasons,
*maxheaderlen* defaults to ``0``, so if you want a different value you
must override it explicitly (the value specified for *max_line_length* in
the policy will be ignored by this method). The *policy* argument may be
used to override the default policy obtained from the message instance.
This can be used to control some of the formatting produced by the
method, since the specified *policy* will be passed to the ``Generator``.
Flattening the message may trigger changes to the :class:`Message` if
defaults need to be filled in to complete the transformation to a string
(for example, MIME boundaries may be generated or modified).
Note that this method is provided as a convenience and may not always
format the message the way you want. For example, by default it does
not do the mangling of lines that begin with ``From`` that is
required by the unix mbox format. For more flexibility, instantiate a
:class:`~email.generator.Generator` instance and use its
:meth:`~email.generator.Generator.flatten` method directly. For example::
from io import StringIO
from email.generator import Generator
fp = StringIO()
g = Generator(fp, mangle_from_=True, maxheaderlen=60)
g.flatten(msg)
text = fp.getvalue()
If the message object contains binary data that is not encoded according
to RFC standards, the non-compliant data will be replaced by unicode
"unknown character" code points. (See also :meth:`.as_bytes` and
:class:`~email.generator.BytesGenerator`.)
.. versionchanged:: 3.4 the *policy* keyword argument was added.
.. method:: __str__()
Equivalent to :meth:`.as_string()`. Allows ``str(msg)`` to produce a
string containing the formatted message.
.. method:: as_bytes(unixfrom=False, policy=None)
Return the entire message flattened as a bytes object. When optional
*unixfrom* is true, the envelope header is included in the returned
string. *unixfrom* defaults to ``False``. The *policy* argument may be
used to override the default policy obtained from the message instance.
This can be used to control some of the formatting produced by the
method, since the specified *policy* will be passed to the
``BytesGenerator``.
Flattening the message may trigger changes to the :class:`Message` if
defaults need to be filled in to complete the transformation to a string
(for example, MIME boundaries may be generated or modified).
Note that this method is provided as a convenience and may not always
format the message the way you want. For example, by default it does
not do the mangling of lines that begin with ``From`` that is
required by the unix mbox format. For more flexibility, instantiate a
:class:`~email.generator.BytesGenerator` instance and use its
:meth:`~email.generator.BytesGenerator.flatten` method directly.
For example::
from io import BytesIO
from email.generator import BytesGenerator
fp = BytesIO()
g = BytesGenerator(fp, mangle_from_=True, maxheaderlen=60)
g.flatten(msg)
text = fp.getvalue()
.. versionadded:: 3.4
.. method:: __bytes__()
Equivalent to :meth:`.as_bytes()`. Allows ``bytes(msg)`` to produce a
bytes object containing the formatted message.
.. versionadded:: 3.4
.. method:: is_multipart()
Return ``True`` if the message's payload is a list of
sub-\ :class:`Message` objects, otherwise return ``False``. When
:meth:`is_multipart` returns ``False``, the payload should be a string
object (which might be a CTE encoded binary payload). (Note that
:meth:`is_multipart` returning ``True`` does not necessarily mean that
"msg.get_content_maintype() == 'multipart'" will return the ``True``.
For example, ``is_multipart`` will return ``True`` when the
:class:`Message` is of type ``message/rfc822``.)
.. method:: set_unixfrom(unixfrom)
Set the message's envelope header to *unixfrom*, which should be a string.
.. method:: get_unixfrom()
Return the message's envelope header. Defaults to ``None`` if the
envelope header was never set.
.. method:: attach(payload)
Add the given *payload* to the current payload, which must be ``None`` or
a list of :class:`Message` objects before the call. After the call, the
payload will always be a list of :class:`Message` objects. If you want to
set the payload to a scalar object (e.g. a string), use
:meth:`set_payload` instead.
This is a legacy method. On the
:class:`~email.emailmessage.EmailMessage` class its functionality is
replaced by :meth:`~email.message.EmailMessage.set_content` and the
related ``make`` and ``add`` methods.
.. method:: get_payload(i=None, decode=False)
Return the current payload, which will be a list of
:class:`Message` objects when :meth:`is_multipart` is ``True``, or a
string when :meth:`is_multipart` is ``False``. If the payload is a list
and you mutate the list object, you modify the message's payload in place.
With optional argument *i*, :meth:`get_payload` will return the *i*-th
element of the payload, counting from zero, if :meth:`is_multipart` is
``True``. An :exc:`IndexError` will be raised if *i* is less than 0 or
greater than or equal to the number of items in the payload. If the
payload is a string (i.e. :meth:`is_multipart` is ``False``) and *i* is
given, a :exc:`TypeError` is raised.
Optional *decode* is a flag indicating whether the payload should be
decoded or not, according to the :mailheader:`Content-Transfer-Encoding`
header. When ``True`` and the message is not a multipart, the payload will
be decoded if this header's value is ``quoted-printable`` or ``base64``.
If some other encoding is used, or :mailheader:`Content-Transfer-Encoding`
header is missing, the payload is
returned as-is (undecoded). In all cases the returned value is binary
data. If the message is a multipart and the *decode* flag is ``True``,
then ``None`` is returned. If the payload is base64 and it was not
perfectly formed (missing padding, characters outside the base64
alphabet), then an appropriate defect will be added to the message's
defect property (:class:`~email.errors.InvalidBase64PaddingDefect` or
:class:`~email.errors.InvalidBase64CharactersDefect`, respectively).
When *decode* is ``False`` (the default) the body is returned as a string
without decoding the :mailheader:`Content-Transfer-Encoding`. However,
for a :mailheader:`Content-Transfer-Encoding` of 8bit, an attempt is made
to decode the original bytes using the ``charset`` specified by the
:mailheader:`Content-Type` header, using the ``replace`` error handler.
If no ``charset`` is specified, or if the ``charset`` given is not
recognized by the email package, the body is decoded using the default
ASCII charset.
This is a legacy method. On the
:class:`~email.emailmessage.EmailMessage` class its functionality is
replaced by :meth:`~email.message.EmailMessage.get_content` and
:meth:`~email.message.EmailMessage.iter_parts`.
.. method:: set_payload(payload, charset=None)
Set the entire message object's payload to *payload*. It is the client's
responsibility to ensure the payload invariants. Optional *charset* sets
the message's default character set; see :meth:`set_charset` for details.
This is a legacy method. On the
:class:`~email.emailmessage.EmailMessage` class its functionality is
replaced by :meth:`~email.message.EmailMessage.set_content`.
.. method:: set_charset(charset)
Set the character set of the payload to *charset*, which can either be a
:class:`~email.charset.Charset` instance (see :mod:`email.charset`), a
string naming a character set, or ``None``. If it is a string, it will
be converted to a :class:`~email.charset.Charset` instance. If *charset*
is ``None``, the ``charset`` parameter will be removed from the
:mailheader:`Content-Type` header (the message will not be otherwise
modified). Anything else will generate a :exc:`TypeError`.
If there is no existing :mailheader:`MIME-Version` header one will be
added. If there is no existing :mailheader:`Content-Type` header, one
will be added with a value of :mimetype:`text/plain`. Whether the
:mailheader:`Content-Type` header already exists or not, its ``charset``
parameter will be set to *charset.output_charset*. If
*charset.input_charset* and *charset.output_charset* differ, the payload
will be re-encoded to the *output_charset*. If there is no existing
:mailheader:`Content-Transfer-Encoding` header, then the payload will be
transfer-encoded, if needed, using the specified
:class:`~email.charset.Charset`, and a header with the appropriate value
will be added. If a :mailheader:`Content-Transfer-Encoding` header
already exists, the payload is assumed to already be correctly encoded
using that :mailheader:`Content-Transfer-Encoding` and is not modified.
This is a legacy method. On the
:class:`~email.emailmessage.EmailMessage` class its functionality is
replaced by the *charset* parameter of the
:meth:`email.emailmessage.EmailMessage.set_content` method.
.. method:: get_charset()
Return the :class:`~email.charset.Charset` instance associated with the
message's payload.
This is a legacy method. On the
:class:`~email.emailmessage.EmailMessage` class it always returns
``None``.
The following methods implement a mapping-like interface for accessing the
message's :rfc:`2822` headers. Note that there are some semantic differences
between these methods and a normal mapping (i.e. dictionary) interface. For
example, in a dictionary there are no duplicate keys, but here there may be
duplicate message headers. Also, in dictionaries there is no guaranteed
order to the keys returned by :meth:`keys`, but in a :class:`Message` object,
headers are always returned in the order they appeared in the original
message, or were added to the message later. Any header deleted and then
re-added are always appended to the end of the header list.
These semantic differences are intentional and are biased toward maximal
convenience.
Note that in all cases, any envelope header present in the message is not
included in the mapping interface.
In a model generated from bytes, any header values that (in contravention of
the RFCs) contain non-ASCII bytes will, when retrieved through this
interface, be represented as :class:`~email.header.Header` objects with
a charset of `unknown-8bit`.
.. method:: __len__()
Return the total number of headers, including duplicates.
.. method:: __contains__(name)
Return true if the message object has a field named *name*. Matching is
done case-insensitively and *name* should not include the trailing colon.
Used for the ``in`` operator, e.g.::
if 'message-id' in myMessage:
print('Message-ID:', myMessage['message-id'])
.. method:: __getitem__(name)
Return the value of the named header field. *name* should not include the
colon field separator. If the header is missing, ``None`` is returned; a
:exc:`KeyError` is never raised.
Note that if the named field appears more than once in the message's
headers, exactly which of those field values will be returned is
undefined. Use the :meth:`get_all` method to get the values of all the
extant named headers.
.. method:: __setitem__(name, val)
Add a header to the message with field name *name* and value *val*. The
field is appended to the end of the message's existing fields.
Note that this does *not* overwrite or delete any existing header with the same
name. If you want to ensure that the new header is the only one present in the
message with field name *name*, delete the field first, e.g.::
del msg['subject']
msg['subject'] = 'Python roolz!'
.. method:: __delitem__(name)
Delete all occurrences of the field with name *name* from the message's
headers. No exception is raised if the named field isn't present in the
headers.
.. method:: keys()
Return a list of all the message's header field names.
.. method:: values()
Return a list of all the message's field values.
.. method:: items()
Return a list of 2-tuples containing all the message's field headers and
values.
.. method:: get(name, failobj=None)
Return the value of the named header field. This is identical to
:meth:`__getitem__` except that optional *failobj* is returned if the
named header is missing (defaults to ``None``).
Here are some additional useful methods:
.. method:: get_all(name, failobj=None)
Return a list of all the values for the field named *name*. If there are
no such named headers in the message, *failobj* is returned (defaults to
``None``).
.. method:: add_header(_name, _value, **_params)
Extended header setting. This method is similar to :meth:`__setitem__`
except that additional header parameters can be provided as keyword
arguments. *_name* is the header field to add and *_value* is the
*primary* value for the header.
For each item in the keyword argument dictionary *_params*, the key is
taken as the parameter name, with underscores converted to dashes (since
dashes are illegal in Python identifiers). Normally, the parameter will
be added as ``key="value"`` unless the value is ``None``, in which case
only the key will be added. If the value contains non-ASCII characters,
it can be specified as a three tuple in the format
``(CHARSET, LANGUAGE, VALUE)``, where ``CHARSET`` is a string naming the
charset to be used to encode the value, ``LANGUAGE`` can usually be set
to ``None`` or the empty string (see :rfc:`2231` for other possibilities),
and ``VALUE`` is the string value containing non-ASCII code points. If
a three tuple is not passed and the value contains non-ASCII characters,
it is automatically encoded in :rfc:`2231` format using a ``CHARSET``
of ``utf-8`` and a ``LANGUAGE`` of ``None``.
Here's an example::
msg.add_header('Content-Disposition', 'attachment', filename='bud.gif')
This will add a header that looks like ::
Content-Disposition: attachment; filename="bud.gif"
An example with non-ASCII characters::
msg.add_header('Content-Disposition', 'attachment',
filename=('iso-8859-1', '', 'Fußballer.ppt'))
Which produces ::
Content-Disposition: attachment; filename*="iso-8859-1''Fu%DFballer.ppt"
.. method:: replace_header(_name, _value)
Replace a header. Replace the first header found in the message that
matches *_name*, retaining header order and field name case. If no
matching header was found, a :exc:`KeyError` is raised.
.. method:: get_content_type()
Return the message's content type. The returned string is coerced to
lower case of the form :mimetype:`maintype/subtype`. If there was no
:mailheader:`Content-Type` header in the message the default type as given
by :meth:`get_default_type` will be returned. Since according to
:rfc:`2045`, messages always have a default type, :meth:`get_content_type`
will always return a value.
:rfc:`2045` defines a message's default type to be :mimetype:`text/plain`
unless it appears inside a :mimetype:`multipart/digest` container, in
which case it would be :mimetype:`message/rfc822`. If the
:mailheader:`Content-Type` header has an invalid type specification,
:rfc:`2045` mandates that the default type be :mimetype:`text/plain`.
.. method:: get_content_maintype()
Return the message's main content type. This is the :mimetype:`maintype`
part of the string returned by :meth:`get_content_type`.
.. method:: get_content_subtype()
Return the message's sub-content type. This is the :mimetype:`subtype`
part of the string returned by :meth:`get_content_type`.
.. method:: get_default_type()
Return the default content type. Most messages have a default content
type of :mimetype:`text/plain`, except for messages that are subparts of
:mimetype:`multipart/digest` containers. Such subparts have a default
content type of :mimetype:`message/rfc822`.
.. method:: set_default_type(ctype)
Set the default content type. *ctype* should either be
:mimetype:`text/plain` or :mimetype:`message/rfc822`, although this is not
enforced. The default content type is not stored in the
:mailheader:`Content-Type` header.
.. method:: get_params(failobj=None, header='content-type', unquote=True)
Return the message's :mailheader:`Content-Type` parameters, as a list.
The elements of the returned list are 2-tuples of key/value pairs, as
split on the ``'='`` sign. The left hand side of the ``'='`` is the key,
while the right hand side is the value. If there is no ``'='`` sign in
the parameter the value is the empty string, otherwise the value is as
described in :meth:`get_param` and is unquoted if optional *unquote* is
``True`` (the default).
Optional *failobj* is the object to return if there is no
:mailheader:`Content-Type` header. Optional *header* is the header to
search instead of :mailheader:`Content-Type`.
This is a legacy method. On the
:class:`~email.emailmessage.EmailMessage` class its functionality is
replaced by the *params* property of the individual header objects
returned by the header access methods.
.. method:: get_param(param, failobj=None, header='content-type', unquote=True)
Return the value of the :mailheader:`Content-Type` header's parameter
*param* as a string. If the message has no :mailheader:`Content-Type`
header or if there is no such parameter, then *failobj* is returned
(defaults to ``None``).
Optional *header* if given, specifies the message header to use instead of
:mailheader:`Content-Type`.
Parameter keys are always compared case insensitively. The return value
can either be a string, or a 3-tuple if the parameter was :rfc:`2231`
encoded. When it's a 3-tuple, the elements of the value are of the form
``(CHARSET, LANGUAGE, VALUE)``. Note that both ``CHARSET`` and
``LANGUAGE`` can be ``None``, in which case you should consider ``VALUE``
to be encoded in the ``us-ascii`` charset. You can usually ignore
``LANGUAGE``.
If your application doesn't care whether the parameter was encoded as in
:rfc:`2231`, you can collapse the parameter value by calling
:func:`email.utils.collapse_rfc2231_value`, passing in the return value
from :meth:`get_param`. This will return a suitably decoded Unicode
string when the value is a tuple, or the original string unquoted if it
isn't. For example::
rawparam = msg.get_param('foo')
param = email.utils.collapse_rfc2231_value(rawparam)
In any case, the parameter value (either the returned string, or the
``VALUE`` item in the 3-tuple) is always unquoted, unless *unquote* is set
to ``False``.
This is a legacy method. On the
:class:`~email.emailmessage.EmailMessage` class its functionality is
replaced by the *params* property of the individual header objects
returned by the header access methods.
.. method:: set_param(param, value, header='Content-Type', requote=True, \
charset=None, language='', replace=False)
Set a parameter in the :mailheader:`Content-Type` header. If the
parameter already exists in the header, its value will be replaced with
*value*. If the :mailheader:`Content-Type` header as not yet been defined
for this message, it will be set to :mimetype:`text/plain` and the new
parameter value will be appended as per :rfc:`2045`.
Optional *header* specifies an alternative header to
:mailheader:`Content-Type`, and all parameters will be quoted as necessary
unless optional *requote* is ``False`` (the default is ``True``).
If optional *charset* is specified, the parameter will be encoded
according to :rfc:`2231`. Optional *language* specifies the RFC 2231
language, defaulting to the empty string. Both *charset* and *language*
should be strings.
If *replace* is ``False`` (the default) the header is moved to the
end of the list of headers. If *replace* is ``True``, the header
will be updated in place.
.. versionchanged:: 3.4 ``replace`` keyword was added.
.. method:: del_param(param, header='content-type', requote=True)
Remove the given parameter completely from the :mailheader:`Content-Type`
header. The header will be re-written in place without the parameter or
its value. All values will be quoted as necessary unless *requote* is
``False`` (the default is ``True``). Optional *header* specifies an
alternative to :mailheader:`Content-Type`.
.. method:: set_type(type, header='Content-Type', requote=True)
Set the main type and subtype for the :mailheader:`Content-Type`
header. *type* must be a string in the form :mimetype:`maintype/subtype`,
otherwise a :exc:`ValueError` is raised.
This method replaces the :mailheader:`Content-Type` header, keeping all
the parameters in place. If *requote* is ``False``, this leaves the
existing header's quoting as is, otherwise the parameters will be quoted
(the default).
An alternative header can be specified in the *header* argument. When the
:mailheader:`Content-Type` header is set a :mailheader:`MIME-Version`
header is also added.
This is a legacy method. On the
:class:`~email.emailmessage.EmailMessage` class its functionality is
replaced by the ``make_`` and ``add_`` methods.
.. method:: get_filename(failobj=None)
Return the value of the ``filename`` parameter of the
:mailheader:`Content-Disposition` header of the message. If the header
does not have a ``filename`` parameter, this method falls back to looking
for the ``name`` parameter on the :mailheader:`Content-Type` header. If
neither is found, or the header is missing, then *failobj* is returned.
The returned string will always be unquoted as per
:func:`email.utils.unquote`.
.. method:: get_boundary(failobj=None)
Return the value of the ``boundary`` parameter of the
:mailheader:`Content-Type` header of the message, or *failobj* if either
the header is missing, or has no ``boundary`` parameter. The returned
string will always be unquoted as per :func:`email.utils.unquote`.
.. method:: set_boundary(boundary)
Set the ``boundary`` parameter of the :mailheader:`Content-Type` header to
*boundary*. :meth:`set_boundary` will always quote *boundary* if
necessary. A :exc:`~email.errors.HeaderParseError` is raised if the
message object has no :mailheader:`Content-Type` header.
Note that using this method is subtly different than deleting the old
:mailheader:`Content-Type` header and adding a new one with the new
boundary via :meth:`add_header`, because :meth:`set_boundary` preserves
the order of the :mailheader:`Content-Type` header in the list of
headers. However, it does *not* preserve any continuation lines which may
have been present in the original :mailheader:`Content-Type` header.
.. method:: get_content_charset(failobj=None)
Return the ``charset`` parameter of the :mailheader:`Content-Type` header,
coerced to lower case. If there is no :mailheader:`Content-Type` header, or if
that header has no ``charset`` parameter, *failobj* is returned.
Note that this method differs from :meth:`get_charset` which returns the
:class:`~email.charset.Charset` instance for the default encoding of the message body.
.. method:: get_charsets(failobj=None)
Return a list containing the character set names in the message. If the
message is a :mimetype:`multipart`, then the list will contain one element
for each subpart in the payload, otherwise, it will be a list of length 1.
Each item in the list will be a string which is the value of the
``charset`` parameter in the :mailheader:`Content-Type` header for the
represented subpart. However, if the subpart has no
:mailheader:`Content-Type` header, no ``charset`` parameter, or is not of
the :mimetype:`text` main MIME type, then that item in the returned list
will be *failobj*.
.. method:: get_content_disposition()
Return the lowercased value (without parameters) of the message's
:mailheader:`Content-Disposition` header if it has one, or ``None``. The
possible values for this method are *inline*, *attachment* or ``None``
if the message follows :rfc:`2183`.
.. versionadded:: 3.5
.. method:: walk()
The :meth:`walk` method is an all-purpose generator which can be used to
iterate over all the parts and subparts of a message object tree, in
depth-first traversal order. You will typically use :meth:`walk` as the
iterator in a ``for`` loop; each iteration returns the next subpart.
Here's an example that prints the MIME type of every part of a multipart
message structure:
.. testsetup::
>>> from email import message_from_binary_file
>>> with open('Lib/test/test_email/data/msg_16.txt', 'rb') as f:
... msg = message_from_binary_file(f)
>>> from email.iterators import _structure
.. doctest::
>>> for part in msg.walk():
... print(part.get_content_type())
multipart/report
text/plain
message/delivery-status
text/plain
text/plain
message/rfc822
text/plain
``walk`` iterates over the subparts of any part where
:meth:`is_multipart` returns ``True``, even though
``msg.get_content_maintype() == 'multipart'`` may return ``False``. We
can see this in our example by making use of the ``_structure`` debug
helper function:
.. doctest::
>>> for part in msg.walk():
... print(part.get_content_maintype() == 'multipart'),
... part.is_multipart())
True True
False False
False True
False False
False False
False True
False False
>>> _structure(msg)
multipart/report
text/plain
message/delivery-status
text/plain
text/plain
message/rfc822
text/plain
Here the ``message`` parts are not ``multiparts``, but they do contain
subparts. ``is_multipart()`` returns ``True`` and ``walk`` descends
into the subparts.
:class:`Message` objects can also optionally contain two instance attributes,
which can be used when generating the plain text of a MIME message.
.. attribute:: preamble
The format of a MIME document allows for some text between the blank line
following the headers, and the first multipart boundary string. Normally,
this text is never visible in a MIME-aware mail reader because it falls
outside the standard MIME armor. However, when viewing the raw text of
the message, or when viewing the message in a non-MIME aware reader, this
text can become visible.
The *preamble* attribute contains this leading extra-armor text for MIME
documents. When the :class:`~email.parser.Parser` discovers some text
after the headers but before the first boundary string, it assigns this
text to the message's *preamble* attribute. When the
:class:`~email.generator.Generator` is writing out the plain text
representation of a MIME message, and it finds the
message has a *preamble* attribute, it will write this text in the area
between the headers and the first boundary. See :mod:`email.parser` and
:mod:`email.generator` for details.
Note that if the message object has no preamble, the *preamble* attribute
will be ``None``.
.. attribute:: epilogue
The *epilogue* attribute acts the same way as the *preamble* attribute,
except that it contains text that appears between the last boundary and
the end of the message.
You do not need to set the epilogue to the empty string in order for the
:class:`~email.generator.Generator` to print a newline at the end of the
file.
.. attribute:: defects
The *defects* attribute contains a list of all the problems found when
parsing this message. See :mod:`email.errors` for a detailed description
of the possible parsing defects.

198
third_party/python/Doc/library/email.contentmanager.rst vendored Normal file
View file

@ -0,0 +1,198 @@
:mod:`email.contentmanager`: Managing MIME Content
--------------------------------------------------
.. module:: email.contentmanager
:synopsis: Storing and Retrieving Content from MIME Parts
.. moduleauthor:: R. David Murray <rdmurray@bitdance.com>
.. sectionauthor:: R. David Murray <rdmurray@bitdance.com>
**Source code:** :source:`Lib/email/contentmanager.py`
------------
.. versionadded:: 3.6 [1]_
.. class:: ContentManager()
Base class for content managers. Provides the standard registry mechanisms
to register converters between MIME content and other representations, as
well as the ``get_content`` and ``set_content`` dispatch methods.
.. method:: get_content(msg, *args, **kw)
Look up a handler function based on the ``mimetype`` of *msg* (see next
paragraph), call it, passing through all arguments, and return the result
of the call. The expectation is that the handler will extract the
payload from *msg* and return an object that encodes information about
the extracted data.
To find the handler, look for the following keys in the registry,
stopping with the first one found:
* the string representing the full MIME type (``maintype/subtype``)
* the string representing the ``maintype``
* the empty string
If none of these keys produce a handler, raise a :exc:`KeyError` for the
full MIME type.
.. method:: set_content(msg, obj, *args, **kw)
If the ``maintype`` is ``multipart``, raise a :exc:`TypeError`; otherwise
look up a handler function based on the type of *obj* (see next
paragraph), call :meth:`~email.message.EmailMessage.clear_content` on the
*msg*, and call the handler function, passing through all arguments. The
expectation is that the handler will transform and store *obj* into
*msg*, possibly making other changes to *msg* as well, such as adding
various MIME headers to encode information needed to interpret the stored
data.
To find the handler, obtain the type of *obj* (``typ = type(obj)``), and
look for the following keys in the registry, stopping with the first one
found:
* the type itself (``typ``)
* the type's fully qualified name (``typ.__module__ + '.' +
typ.__qualname__``).
* the type's qualname (``typ.__qualname__``)
* the type's name (``typ.__name__``).
If none of the above match, repeat all of the checks above for each of
the types in the :term:`MRO` (``typ.__mro__``). Finally, if no other key
yields a handler, check for a handler for the key ``None``. If there is
no handler for ``None``, raise a :exc:`KeyError` for the fully
qualified name of the type.
Also add a :mailheader:`MIME-Version` header if one is not present (see
also :class:`.MIMEPart`).
.. method:: add_get_handler(key, handler)
Record the function *handler* as the handler for *key*. For the possible
values of *key*, see :meth:`get_content`.
.. method:: add_set_handler(typekey, handler)
Record *handler* as the function to call when an object of a type
matching *typekey* is passed to :meth:`set_content`. For the possible
values of *typekey*, see :meth:`set_content`.
Content Manager Instances
~~~~~~~~~~~~~~~~~~~~~~~~~
Currently the email package provides only one concrete content manager,
:data:`raw_data_manager`, although more may be added in the future.
:data:`raw_data_manager` is the
:attr:`~email.policy.EmailPolicy.content_manager` provided by
:attr:`~email.policy.EmailPolicy` and its derivatives.
.. data:: raw_data_manager
This content manager provides only a minimum interface beyond that provided
by :class:`~email.message.Message` itself: it deals only with text, raw
byte strings, and :class:`~email.message.Message` objects. Nevertheless, it
provides significant advantages compared to the base API: ``get_content`` on
a text part will return a unicode string without the application needing to
manually decode it, ``set_content`` provides a rich set of options for
controlling the headers added to a part and controlling the content transfer
encoding, and it enables the use of the various ``add_`` methods, thereby
simplifying the creation of multipart messages.
.. method:: get_content(msg, errors='replace')
Return the payload of the part as either a string (for ``text`` parts), an
:class:`~email.message.EmailMessage` object (for ``message/rfc822``
parts), or a ``bytes`` object (for all other non-multipart types). Raise
a :exc:`KeyError` if called on a ``multipart``. If the part is a
``text`` part and *errors* is specified, use it as the error handler when
decoding the payload to unicode. The default error handler is
``replace``.
.. method:: set_content(msg, <'str'>, subtype="plain", charset='utf-8' \
cte=None, \
disposition=None, filename=None, cid=None, \
params=None, headers=None)
set_content(msg, <'bytes'>, maintype, subtype, cte="base64", \
disposition=None, filename=None, cid=None, \
params=None, headers=None)
set_content(msg, <'EmailMessage'>, cte=None, \
disposition=None, filename=None, cid=None, \
params=None, headers=None)
Add headers and payload to *msg*:
Add a :mailheader:`Content-Type` header with a ``maintype/subtype``
value.
* For ``str``, set the MIME ``maintype`` to ``text``, and set the
subtype to *subtype* if it is specified, or ``plain`` if it is not.
* For ``bytes``, use the specified *maintype* and *subtype*, or
raise a :exc:`TypeError` if they are not specified.
* For :class:`~email.message.EmailMessage` objects, set the maintype
to ``message``, and set the subtype to *subtype* if it is
specified or ``rfc822`` if it is not. If *subtype* is
``partial``, raise an error (``bytes`` objects must be used to
construct ``message/partial`` parts).
If *charset* is provided (which is valid only for ``str``), encode the
string to bytes using the specified character set. The default is
``utf-8``. If the specified *charset* is a known alias for a standard
MIME charset name, use the standard charset instead.
If *cte* is set, encode the payload using the specified content transfer
encoding, and set the :mailheader:`Content-Transfer-Encoding` header to
that value. Possible values for *cte* are ``quoted-printable``,
``base64``, ``7bit``, ``8bit``, and ``binary``. If the input cannot be
encoded in the specified encoding (for example, specifying a *cte* of
``7bit`` for an input that contains non-ASCII values), raise a
:exc:`ValueError`.
* For ``str`` objects, if *cte* is not set use heuristics to
determine the most compact encoding.
* For :class:`~email.message.EmailMessage`, per :rfc:`2046`, raise
an error if a *cte* of ``quoted-printable`` or ``base64`` is
requested for *subtype* ``rfc822``, and for any *cte* other than
``7bit`` for *subtype* ``external-body``. For
``message/rfc822``, use ``8bit`` if *cte* is not specified. For
all other values of *subtype*, use ``7bit``.
.. note:: A *cte* of ``binary`` does not actually work correctly yet.
The ``EmailMessage`` object as modified by ``set_content`` is
correct, but :class:`~email.generator.BytesGenerator` does not
serialize it correctly.
If *disposition* is set, use it as the value of the
:mailheader:`Content-Disposition` header. If not specified, and
*filename* is specified, add the header with the value ``attachment``.
If *disposition* is not specified and *filename* is also not specified,
do not add the header. The only valid values for *disposition* are
``attachment`` and ``inline``.
If *filename* is specified, use it as the value of the ``filename``
parameter of the :mailheader:`Content-Disposition` header.
If *cid* is specified, add a :mailheader:`Content-ID` header with
*cid* as its value.
If *params* is specified, iterate its ``items`` method and use the
resulting ``(key, value)`` pairs to set additional parameters on the
:mailheader:`Content-Type` header.
If *headers* is specified and is a list of strings of the form
``headername: headervalue`` or a list of ``header`` objects
(distinguished from strings by having a ``name`` attribute), add the
headers to *msg*.
.. rubric:: Footnotes
.. [1] Originally added in 3.4 as a :term:`provisional module <provisional
package>`

70
third_party/python/Doc/library/email.encoders.rst vendored Normal file
View file

@ -0,0 +1,70 @@
:mod:`email.encoders`: Encoders
-------------------------------
.. module:: email.encoders
:synopsis: Encoders for email message payloads.
**Source code:** :source:`Lib/email/encoders.py`
--------------
This module is part of the legacy (``Compat32``) email API. In the
new API the functionality is provided by the *cte* parameter of
the :meth:`~email.message.EmailMessage.set_content` method.
The remaining text in this section is the original documentation of the module.
When creating :class:`~email.message.Message` objects from scratch, you often
need to encode the payloads for transport through compliant mail servers. This
is especially true for :mimetype:`image/\*` and :mimetype:`text/\*` type messages
containing binary data.
The :mod:`email` package provides some convenient encodings in its
:mod:`encoders` module. These encoders are actually used by the
:class:`~email.mime.audio.MIMEAudio` and :class:`~email.mime.image.MIMEImage`
class constructors to provide default encodings. All encoder functions take
exactly one argument, the message object to encode. They usually extract the
payload, encode it, and reset the payload to this newly encoded value. They
should also set the :mailheader:`Content-Transfer-Encoding` header as appropriate.
Note that these functions are not meaningful for a multipart message. They
must be applied to individual subparts instead, and will raise a
:exc:`TypeError` if passed a message whose type is multipart.
Here are the encoding functions provided:
.. function:: encode_quopri(msg)
Encodes the payload into quoted-printable form and sets the
:mailheader:`Content-Transfer-Encoding` header to ``quoted-printable`` [#]_.
This is a good encoding to use when most of your payload is normal printable
data, but contains a few unprintable characters.
.. function:: encode_base64(msg)
Encodes the payload into base64 form and sets the
:mailheader:`Content-Transfer-Encoding` header to ``base64``. This is a good
encoding to use when most of your payload is unprintable data since it is a more
compact form than quoted-printable. The drawback of base64 encoding is that it
renders the text non-human readable.
.. function:: encode_7or8bit(msg)
This doesn't actually modify the message's payload, but it does set the
:mailheader:`Content-Transfer-Encoding` header to either ``7bit`` or ``8bit`` as
appropriate, based on the payload data.
.. function:: encode_noop(msg)
This does nothing; it doesn't even set the
:mailheader:`Content-Transfer-Encoding` header.
.. rubric:: Footnotes
.. [#] Note that encoding with :meth:`encode_quopri` also encodes all tabs and space
characters in the data.

114
third_party/python/Doc/library/email.errors.rst vendored Normal file
View file

@ -0,0 +1,114 @@
:mod:`email.errors`: Exception and Defect classes
-------------------------------------------------
.. module:: email.errors
:synopsis: The exception classes used by the email package.
**Source code:** :source:`Lib/email/errors.py`
--------------
The following exception classes are defined in the :mod:`email.errors` module:
.. exception:: MessageError()
This is the base class for all exceptions that the :mod:`email` package can
raise. It is derived from the standard :exc:`Exception` class and defines no
additional methods.
.. exception:: MessageParseError()
This is the base class for exceptions raised by the
:class:`~email.parser.Parser` class. It is derived from
:exc:`MessageError`. This class is also used internally by the parser used
by :mod:`~email.headerregistry`.
.. exception:: HeaderParseError()
Raised under some error conditions when parsing the :rfc:`5322` headers of a
message, this class is derived from :exc:`MessageParseError`. The
:meth:`~email.message.EmailMessage.set_boundary` method will raise this
error if the content type is unknown when the method is called.
:class:`~email.header.Header` may raise this error for certain base64
decoding errors, and when an attempt is made to create a header that appears
to contain an embedded header (that is, there is what is supposed to be a
continuation line that has no leading whitespace and looks like a header).
.. exception:: BoundaryError()
Deprecated and no longer used.
.. exception:: MultipartConversionError()
Raised when a payload is added to a :class:`~email.message.Message` object
using :meth:`add_payload`, but the payload is already a scalar and the
message's :mailheader:`Content-Type` main type is not either
:mimetype:`multipart` or missing. :exc:`MultipartConversionError` multiply
inherits from :exc:`MessageError` and the built-in :exc:`TypeError`.
Since :meth:`Message.add_payload` is deprecated, this exception is rarely
raised in practice. However the exception may also be raised if the
:meth:`~email.message.Message.attach`
method is called on an instance of a class derived from
:class:`~email.mime.nonmultipart.MIMENonMultipart` (e.g.
:class:`~email.mime.image.MIMEImage`).
Here is the list of the defects that the :class:`~email.parser.FeedParser`
can find while parsing messages. Note that the defects are added to the message
where the problem was found, so for example, if a message nested inside a
:mimetype:`multipart/alternative` had a malformed header, that nested message
object would have a defect, but the containing messages would not.
All defect classes are subclassed from :class:`email.errors.MessageDefect`.
* :class:`NoBoundaryInMultipartDefect` -- A message claimed to be a multipart,
but had no :mimetype:`boundary` parameter.
* :class:`StartBoundaryNotFoundDefect` -- The start boundary claimed in the
:mailheader:`Content-Type` header was never found.
* :class:`CloseBoundaryNotFoundDefect` -- A start boundary was found, but
no corresponding close boundary was ever found.
.. versionadded:: 3.3
* :class:`FirstHeaderLineIsContinuationDefect` -- The message had a continuation
line as its first header line.
* :class:`MisplacedEnvelopeHeaderDefect` - A "Unix From" header was found in the
middle of a header block.
* :class:`MissingHeaderBodySeparatorDefect` - A line was found while parsing
headers that had no leading white space but contained no ':'. Parsing
continues assuming that the line represents the first line of the body.
.. versionadded:: 3.3
* :class:`MalformedHeaderDefect` -- A header was found that was missing a colon,
or was otherwise malformed.
.. deprecated:: 3.3
This defect has not been used for several Python versions.
* :class:`MultipartInvariantViolationDefect` -- A message claimed to be a
:mimetype:`multipart`, but no subparts were found. Note that when a message
has this defect, its :meth:`~email.message.Message.is_multipart` method may
return false even though its content type claims to be :mimetype:`multipart`.
* :class:`InvalidBase64PaddingDefect` -- When decoding a block of base64
encoded bytes, the padding was not correct. Enough padding is added to
perform the decode, but the resulting decoded bytes may be invalid.
* :class:`InvalidBase64CharactersDefect` -- When decoding a block of base64
encoded bytes, characters outside the base64 alphabet were encountered.
The characters are ignored, but the resulting decoded bytes may be invalid.
* :class:`InvalidBase64LengthDefect` -- When decoding a block of base64 encoded
bytes, the number of non-padding base64 characters was invalid (1 more than
a multiple of 4). The encoded block was kept as-is.

67
third_party/python/Doc/library/email.examples.rst vendored Normal file
View file

@ -0,0 +1,67 @@
.. _email-examples:
:mod:`email`: Examples
----------------------
Here are a few examples of how to use the :mod:`email` package to read, write,
and send simple email messages, as well as more complex MIME messages.
First, let's see how to create and send a simple text message (both the
text content and the addresses may contain unicode characters):
.. literalinclude:: ../includes/email-simple.py
Parsing :rfc:`822` headers can easily be done by the using the classes
from the :mod:`~email.parser` module:
.. literalinclude:: ../includes/email-headers.py
Here's an example of how to send a MIME message containing a bunch of family
pictures that may be residing in a directory:
.. literalinclude:: ../includes/email-mime.py
Here's an example of how to send the entire contents of a directory as an email
message: [1]_
.. literalinclude:: ../includes/email-dir.py
Here's an example of how to unpack a MIME message like the one
above, into a directory of files:
.. literalinclude:: ../includes/email-unpack.py
Here's an example of how to create an HTML message with an alternative plain
text version. To make things a bit more interesting, we include a related
image in the html part, and we save a copy of what we are going to send to
disk, as well as sending it.
.. literalinclude:: ../includes/email-alternative.py
If we were sent the message from the last example, here is one way we could
process it:
.. literalinclude:: ../includes/email-read-alternative.py
Up to the prompt, the output from the above is:
.. code-block:: none
To: Penelope Pussycat <penelope@example.com>, Fabrette Pussycat <fabrette@example.com>
From: Pepé Le Pew <pepe@example.com>
Subject: Ayons asperges pour le déjeuner
Salut!
Cela ressemble à un excellent recipie[1] déjeuner.
.. rubric:: Footnotes
.. [1] Thanks to Matthew Dixon Cowles for the original inspiration and examples.

279
third_party/python/Doc/library/email.generator.rst vendored Normal file
View file

@ -0,0 +1,279 @@
:mod:`email.generator`: Generating MIME documents
-------------------------------------------------
.. module:: email.generator
:synopsis: Generate flat text email messages from a message structure.
**Source code:** :source:`Lib/email/generator.py`
--------------
One of the most common tasks is to generate the flat (serialized) version of
the email message represented by a message object structure. You will need to
do this if you want to send your message via :meth:`smtplib.SMTP.sendmail` or
the :mod:`nntplib` module, or print the message on the console. Taking a
message object structure and producing a serialized representation is the job
of the generator classes.
As with the :mod:`email.parser` module, you aren't limited to the functionality
of the bundled generator; you could write one from scratch yourself. However
the bundled generator knows how to generate most email in a standards-compliant
way, should handle MIME and non-MIME email messages just fine, and is designed
so that the bytes-oriented parsing and generation operations are inverses,
assuming the same non-transforming :mod:`~email.policy` is used for both. That
is, parsing the serialized byte stream via the
:class:`~email.parser.BytesParser` class and then regenerating the serialized
byte stream using :class:`BytesGenerator` should produce output identical to
the input [#]_. (On the other hand, using the generator on an
:class:`~email.message.EmailMessage` constructed by program may result in
changes to the :class:`~email.message.EmailMessage` object as defaults are
filled in.)
The :class:`Generator` class can be used to flatten a message into a text (as
opposed to binary) serialized representation, but since Unicode cannot
represent binary data directly, the message is of necessity transformed into
something that contains only ASCII characters, using the standard email RFC
Content Transfer Encoding techniques for encoding email messages for transport
over channels that are not "8 bit clean".
.. class:: BytesGenerator(outfp, mangle_from_=None, maxheaderlen=None, *, \
policy=None)
Return a :class:`BytesGenerator` object that will write any message provided
to the :meth:`flatten` method, or any surrogateescape encoded text provided
to the :meth:`write` method, to the :term:`file-like object` *outfp*.
*outfp* must support a ``write`` method that accepts binary data.
If optional *mangle_from_* is ``True``, put a ``>`` character in front of
any line in the body that starts with the exact string ``"From "``, that is
``From`` followed by a space at the beginning of a line. *mangle_from_*
defaults to the value of the :attr:`~email.policy.Policy.mangle_from_`
setting of the *policy* (which is ``True`` for the
:data:`~email.policy.compat32` policy and ``False`` for all others).
*mangle_from_* is intended for use when messages are stored in unix mbox
format (see :mod:`mailbox` and `WHY THE CONTENT-LENGTH FORMAT IS BAD
<http://www.jwz.org/doc/content-length.html>`_).
If *maxheaderlen* is not ``None``, refold any header lines that are longer
than *maxheaderlen*, or if ``0``, do not rewrap any headers. If
*manheaderlen* is ``None`` (the default), wrap headers and other message
lines according to the *policy* settings.
If *policy* is specified, use that policy to control message generation. If
*policy* is ``None`` (the default), use the policy associated with the
:class:`~email.message.Message` or :class:`~email.message.EmailMessage`
object passed to ``flatten`` to control the message generation. See
:mod:`email.policy` for details on what *policy* controls.
.. versionadded:: 3.2
.. versionchanged:: 3.3 Added the *policy* keyword.
.. versionchanged:: 3.6 The default behavior of the *mangle_from_*
and *maxheaderlen* parameters is to follow the policy.
.. method:: flatten(msg, unixfrom=False, linesep=None)
Print the textual representation of the message object structure rooted
at *msg* to the output file specified when the :class:`BytesGenerator`
instance was created.
If the :mod:`~email.policy` option :attr:`~email.policy.Policy.cte_type`
is ``8bit`` (the default), copy any headers in the original parsed
message that have not been modified to the output with any bytes with the
high bit set reproduced as in the original, and preserve the non-ASCII
:mailheader:`Content-Transfer-Encoding` of any body parts that have them.
If ``cte_type`` is ``7bit``, convert the bytes with the high bit set as
needed using an ASCII-compatible :mailheader:`Content-Transfer-Encoding`.
That is, transform parts with non-ASCII
:mailheader:`Content-Transfer-Encoding`
(:mailheader:`Content-Transfer-Encoding: 8bit`) to an ASCII compatible
:mailheader:`Content-Transfer-Encoding`, and encode RFC-invalid non-ASCII
bytes in headers using the MIME ``unknown-8bit`` character set, thus
rendering them RFC-compliant.
.. XXX: There should be an option that just does the RFC
compliance transformation on headers but leaves CTE 8bit parts alone.
If *unixfrom* is ``True``, print the envelope header delimiter used by
the Unix mailbox format (see :mod:`mailbox`) before the first of the
:rfc:`5322` headers of the root message object. If the root object has
no envelope header, craft a standard one. The default is ``False``.
Note that for subparts, no envelope header is ever printed.
If *linesep* is not ``None``, use it as the separator character between
all the lines of the flattened message. If *linesep* is ``None`` (the
default), use the value specified in the *policy*.
.. XXX: flatten should take a *policy* keyword.
.. method:: clone(fp)
Return an independent clone of this :class:`BytesGenerator` instance with
the exact same option settings, and *fp* as the new *outfp*.
.. method:: write(s)
Encode *s* using the ``ASCII`` codec and the ``surrogateescape`` error
handler, and pass it to the *write* method of the *outfp* passed to the
:class:`BytesGenerator`'s constructor.
As a convenience, :class:`~email.message.EmailMessage` provides the methods
:meth:`~email.message.EmailMessage.as_bytes` and ``bytes(aMessage)`` (a.k.a.
:meth:`~email.message.EmailMessage.__bytes__`), which simplify the generation of
a serialized binary representation of a message object. For more detail, see
:mod:`email.message`.
Because strings cannot represent binary data, the :class:`Generator` class must
convert any binary data in any message it flattens to an ASCII compatible
format, by converting them to an ASCII compatible
:mailheader:`Content-Transfer_Encoding`. Using the terminology of the email
RFCs, you can think of this as :class:`Generator` serializing to an I/O stream
that is not "8 bit clean". In other words, most applications will want
to be using :class:`BytesGenerator`, and not :class:`Generator`.
.. class:: Generator(outfp, mangle_from_=None, maxheaderlen=None, *, \
policy=None)
Return a :class:`Generator` object that will write any message provided
to the :meth:`flatten` method, or any text provided to the :meth:`write`
method, to the :term:`file-like object` *outfp*. *outfp* must support a
``write`` method that accepts string data.
If optional *mangle_from_* is ``True``, put a ``>`` character in front of
any line in the body that starts with the exact string ``"From "``, that is
``From`` followed by a space at the beginning of a line. *mangle_from_*
defaults to the value of the :attr:`~email.policy.Policy.mangle_from_`
setting of the *policy* (which is ``True`` for the
:data:`~email.policy.compat32` policy and ``False`` for all others).
*mangle_from_* is intended for use when messages are stored in unix mbox
format (see :mod:`mailbox` and `WHY THE CONTENT-LENGTH FORMAT IS BAD
<http://www.jwz.org/doc/content-length.html>`_).
If *maxheaderlen* is not ``None``, refold any header lines that are longer
than *maxheaderlen*, or if ``0``, do not rewrap any headers. If
*manheaderlen* is ``None`` (the default), wrap headers and other message
lines according to the *policy* settings.
If *policy* is specified, use that policy to control message generation. If
*policy* is ``None`` (the default), use the policy associated with the
:class:`~email.message.Message` or :class:`~email.message.EmailMessage`
object passed to ``flatten`` to control the message generation. See
:mod:`email.policy` for details on what *policy* controls.
.. versionchanged:: 3.3 Added the *policy* keyword.
.. versionchanged:: 3.6 The default behavior of the *mangle_from_*
and *maxheaderlen* parameters is to follow the policy.
.. method:: flatten(msg, unixfrom=False, linesep=None)
Print the textual representation of the message object structure rooted
at *msg* to the output file specified when the :class:`Generator`
instance was created.
If the :mod:`~email.policy` option :attr:`~email.policy.Policy.cte_type`
is ``8bit``, generate the message as if the option were set to ``7bit``.
(This is required because strings cannot represent non-ASCII bytes.)
Convert any bytes with the high bit set as needed using an
ASCII-compatible :mailheader:`Content-Transfer-Encoding`. That is,
transform parts with non-ASCII :mailheader:`Cotnent-Transfer-Encoding`
(:mailheader:`Content-Transfer-Encoding: 8bit`) to an ASCII compatible
:mailheader:`Content-Transfer-Encoding`, and encode RFC-invalid non-ASCII
bytes in headers using the MIME ``unknown-8bit`` character set, thus
rendering them RFC-compliant.
If *unixfrom* is ``True``, print the envelope header delimiter used by
the Unix mailbox format (see :mod:`mailbox`) before the first of the
:rfc:`5322` headers of the root message object. If the root object has
no envelope header, craft a standard one. The default is ``False``.
Note that for subparts, no envelope header is ever printed.
If *linesep* is not ``None``, use it as the separator character between
all the lines of the flattened message. If *linesep* is ``None`` (the
default), use the value specified in the *policy*.
.. XXX: flatten should take a *policy* keyword.
.. versionchanged:: 3.2
Added support for re-encoding ``8bit`` message bodies, and the
*linesep* argument.
.. method:: clone(fp)
Return an independent clone of this :class:`Generator` instance with the
exact same options, and *fp* as the new *outfp*.
.. method:: write(s)
Write *s* to the *write* method of the *outfp* passed to the
:class:`Generator`'s constructor. This provides just enough file-like
API for :class:`Generator` instances to be used in the :func:`print`
function.
As a convenience, :class:`~email.message.EmailMessage` provides the methods
:meth:`~email.message.EmailMessage.as_string` and ``str(aMessage)`` (a.k.a.
:meth:`~email.message.EmailMessage.__str__`), which simplify the generation of
a formatted string representation of a message object. For more detail, see
:mod:`email.message`.
The :mod:`email.generator` module also provides a derived class,
:class:`DecodedGenerator`, which is like the :class:`Generator` base class,
except that non-\ :mimetype:`text` parts are not serialized, but are instead
represented in the output stream by a string derived from a template filled
in with information about the part.
.. class:: DecodedGenerator(outfp, mangle_from_=None, maxheaderlen=None, \
fmt=None, *, policy=None)
Act like :class:`Generator`, except that for any subpart of the message
passed to :meth:`Generator.flatten`, if the subpart is of main type
:mimetype:`text`, print the decoded payload of the subpart, and if the main
type is not :mimetype:`text`, instead of printing it fill in the string
*fmt* using information from the part and print the resulting
filled-in string.
To fill in *fmt*, execute ``fmt % part_info``, where ``part_info``
is a dictionary composed of the following keys and values:
* ``type`` -- Full MIME type of the non-\ :mimetype:`text` part
* ``maintype`` -- Main MIME type of the non-\ :mimetype:`text` part
* ``subtype`` -- Sub-MIME type of the non-\ :mimetype:`text` part
* ``filename`` -- Filename of the non-\ :mimetype:`text` part
* ``description`` -- Description associated with the non-\ :mimetype:`text` part
* ``encoding`` -- Content transfer encoding of the non-\ :mimetype:`text` part
If *fmt* is ``None``, use the following default *fmt*:
"[Non-text (%(type)s) part of message omitted, filename %(filename)s]"
Optional *_mangle_from_* and *maxheaderlen* are as with the
:class:`Generator` base class.
.. rubric:: Footnotes
.. [#] This statement assumes that you use the appropriate setting for
``unixfrom``, and that there are no :mod:`policy` settings calling for
automatic adjustments (for example,
:attr:`~email.policy.Policy.refold_source` must be ``none``, which is
*not* the default). It is also not 100% true, since if the message
does not conform to the RFC standards occasionally information about the
exact original text is lost during parsing error recovery. It is a goal
to fix these latter edge cases when possible.

205
third_party/python/Doc/library/email.header.rst vendored Normal file
View file

@ -0,0 +1,205 @@
:mod:`email.header`: Internationalized headers
----------------------------------------------
.. module:: email.header
:synopsis: Representing non-ASCII headers
**Source code:** :source:`Lib/email/header.py`
--------------
This module is part of the legacy (``Compat32``) email API. In the current API
encoding and decoding of headers is handled transparently by the
dictionary-like API of the :class:`~email.message.EmailMessage` class. In
addition to uses in legacy code, this module can be useful in applications that
need to completely control the character sets used when encoding headers.
The remaining text in this section is the original documentation of the module.
:rfc:`2822` is the base standard that describes the format of email messages.
It derives from the older :rfc:`822` standard which came into widespread use at
a time when most email was composed of ASCII characters only. :rfc:`2822` is a
specification written assuming email contains only 7-bit ASCII characters.
Of course, as email has been deployed worldwide, it has become
internationalized, such that language specific character sets can now be used in
email messages. The base standard still requires email messages to be
transferred using only 7-bit ASCII characters, so a slew of RFCs have been
written describing how to encode email containing non-ASCII characters into
:rfc:`2822`\ -compliant format. These RFCs include :rfc:`2045`, :rfc:`2046`,
:rfc:`2047`, and :rfc:`2231`. The :mod:`email` package supports these standards
in its :mod:`email.header` and :mod:`email.charset` modules.
If you want to include non-ASCII characters in your email headers, say in the
:mailheader:`Subject` or :mailheader:`To` fields, you should use the
:class:`Header` class and assign the field in the :class:`~email.message.Message`
object to an instance of :class:`Header` instead of using a string for the header
value. Import the :class:`Header` class from the :mod:`email.header` module.
For example::
>>> from email.message import Message
>>> from email.header import Header
>>> msg = Message()
>>> h = Header('p\xf6stal', 'iso-8859-1')
>>> msg['Subject'] = h
>>> msg.as_string()
'Subject: =?iso-8859-1?q?p=F6stal?=\n\n'
Notice here how we wanted the :mailheader:`Subject` field to contain a non-ASCII
character? We did this by creating a :class:`Header` instance and passing in
the character set that the byte string was encoded in. When the subsequent
:class:`~email.message.Message` instance was flattened, the :mailheader:`Subject`
field was properly :rfc:`2047` encoded. MIME-aware mail readers would show this
header using the embedded ISO-8859-1 character.
Here is the :class:`Header` class description:
.. class:: Header(s=None, charset=None, maxlinelen=None, header_name=None, continuation_ws=' ', errors='strict')
Create a MIME-compliant header that can contain strings in different character
sets.
Optional *s* is the initial header value. If ``None`` (the default), the
initial header value is not set. You can later append to the header with
:meth:`append` method calls. *s* may be an instance of :class:`bytes` or
:class:`str`, but see the :meth:`append` documentation for semantics.
Optional *charset* serves two purposes: it has the same meaning as the *charset*
argument to the :meth:`append` method. It also sets the default character set
for all subsequent :meth:`append` calls that omit the *charset* argument. If
*charset* is not provided in the constructor (the default), the ``us-ascii``
character set is used both as *s*'s initial charset and as the default for
subsequent :meth:`append` calls.
The maximum line length can be specified explicitly via *maxlinelen*. For
splitting the first line to a shorter value (to account for the field header
which isn't included in *s*, e.g. :mailheader:`Subject`) pass in the name of the
field in *header_name*. The default *maxlinelen* is 76, and the default value
for *header_name* is ``None``, meaning it is not taken into account for the
first line of a long, split header.
Optional *continuation_ws* must be :rfc:`2822`\ -compliant folding
whitespace, and is usually either a space or a hard tab character. This
character will be prepended to continuation lines. *continuation_ws*
defaults to a single space character.
Optional *errors* is passed straight through to the :meth:`append` method.
.. method:: append(s, charset=None, errors='strict')
Append the string *s* to the MIME header.
Optional *charset*, if given, should be a :class:`~email.charset.Charset`
instance (see :mod:`email.charset`) or the name of a character set, which
will be converted to a :class:`~email.charset.Charset` instance. A value
of ``None`` (the default) means that the *charset* given in the constructor
is used.
*s* may be an instance of :class:`bytes` or :class:`str`. If it is an
instance of :class:`bytes`, then *charset* is the encoding of that byte
string, and a :exc:`UnicodeError` will be raised if the string cannot be
decoded with that character set.
If *s* is an instance of :class:`str`, then *charset* is a hint specifying
the character set of the characters in the string.
In either case, when producing an :rfc:`2822`\ -compliant header using
:rfc:`2047` rules, the string will be encoded using the output codec of
the charset. If the string cannot be encoded using the output codec, a
UnicodeError will be raised.
Optional *errors* is passed as the errors argument to the decode call
if *s* is a byte string.
.. method:: encode(splitchars=';, \\t', maxlinelen=None, linesep='\\n')
Encode a message header into an RFC-compliant format, possibly wrapping
long lines and encapsulating non-ASCII parts in base64 or quoted-printable
encodings.
Optional *splitchars* is a string containing characters which should be
given extra weight by the splitting algorithm during normal header
wrapping. This is in very rough support of :RFC:`2822`\'s 'higher level
syntactic breaks': split points preceded by a splitchar are preferred
during line splitting, with the characters preferred in the order in
which they appear in the string. Space and tab may be included in the
string to indicate whether preference should be given to one over the
other as a split point when other split chars do not appear in the line
being split. Splitchars does not affect :RFC:`2047` encoded lines.
*maxlinelen*, if given, overrides the instance's value for the maximum
line length.
*linesep* specifies the characters used to separate the lines of the
folded header. It defaults to the most useful value for Python
application code (``\n``), but ``\r\n`` can be specified in order
to produce headers with RFC-compliant line separators.
.. versionchanged:: 3.2
Added the *linesep* argument.
The :class:`Header` class also provides a number of methods to support
standard operators and built-in functions.
.. method:: __str__()
Returns an approximation of the :class:`Header` as a string, using an
unlimited line length. All pieces are converted to unicode using the
specified encoding and joined together appropriately. Any pieces with a
charset of ``'unknown-8bit'`` are decoded as ASCII using the ``'replace'``
error handler.
.. versionchanged:: 3.2
Added handling for the ``'unknown-8bit'`` charset.
.. method:: __eq__(other)
This method allows you to compare two :class:`Header` instances for
equality.
.. method:: __ne__(other)
This method allows you to compare two :class:`Header` instances for
inequality.
The :mod:`email.header` module also provides the following convenient functions.
.. function:: decode_header(header)
Decode a message header value without converting the character set. The header
value is in *header*.
This function returns a list of ``(decoded_string, charset)`` pairs containing
each of the decoded parts of the header. *charset* is ``None`` for non-encoded
parts of the header, otherwise a lower case string containing the name of the
character set specified in the encoded string.
Here's an example::
>>> from email.header import decode_header
>>> decode_header('=?iso-8859-1?q?p=F6stal?=')
[(b'p\xf6stal', 'iso-8859-1')]
.. function:: make_header(decoded_seq, maxlinelen=None, header_name=None, continuation_ws=' ')
Create a :class:`Header` instance from a sequence of pairs as returned by
:func:`decode_header`.
:func:`decode_header` takes a header value string and returns a sequence of
pairs of the format ``(decoded_string, charset)`` where *charset* is the name of
the character set.
This function takes one of those sequence of pairs and returns a
:class:`Header` instance. Optional *maxlinelen*, *header_name*, and
*continuation_ws* are as in the :class:`Header` constructor.

455
third_party/python/Doc/library/email.headerregistry.rst vendored Normal file
View file

@ -0,0 +1,455 @@
:mod:`email.headerregistry`: Custom Header Objects
--------------------------------------------------
.. module:: email.headerregistry
:synopsis: Automatic Parsing of headers based on the field name
.. moduleauthor:: R. David Murray <rdmurray@bitdance.com>
.. sectionauthor:: R. David Murray <rdmurray@bitdance.com>
**Source code:** :source:`Lib/email/headerregistry.py`
--------------
.. versionadded:: 3.6 [1]_
Headers are represented by customized subclasses of :class:`str`. The
particular class used to represent a given header is determined by the
:attr:`~email.policy.EmailPolicy.header_factory` of the :mod:`~email.policy` in
effect when the headers are created. This section documents the particular
``header_factory`` implemented by the email package for handling :RFC:`5322`
compliant email messages, which not only provides customized header objects for
various header types, but also provides an extension mechanism for applications
to add their own custom header types.
When using any of the policy objects derived from
:data:`~email.policy.EmailPolicy`, all headers are produced by
:class:`.HeaderRegistry` and have :class:`.BaseHeader` as their last base
class. Each header class has an additional base class that is determined by
the type of the header. For example, many headers have the class
:class:`.UnstructuredHeader` as their other base class. The specialized second
class for a header is determined by the name of the header, using a lookup
table stored in the :class:`.HeaderRegistry`. All of this is managed
transparently for the typical application program, but interfaces are provided
for modifying the default behavior for use by more complex applications.
The sections below first document the header base classes and their attributes,
followed by the API for modifying the behavior of :class:`.HeaderRegistry`, and
finally the support classes used to represent the data parsed from structured
headers.
.. class:: BaseHeader(name, value)
*name* and *value* are passed to ``BaseHeader`` from the
:attr:`~email.policy.EmailPolicy.header_factory` call. The string value of
any header object is the *value* fully decoded to unicode.
This base class defines the following read-only properties:
.. attribute:: name
The name of the header (the portion of the field before the ':'). This
is exactly the value passed in the
:attr:`~email.policy.EmailPolicy.header_factory` call for *name*; that
is, case is preserved.
.. attribute:: defects
A tuple of :exc:`~email.errors.HeaderDefect` instances reporting any
RFC compliance problems found during parsing. The email package tries to
be complete about detecting compliance issues. See the :mod:`~email.errors`
module for a discussion of the types of defects that may be reported.
.. attribute:: max_count
The maximum number of headers of this type that can have the same
``name``. A value of ``None`` means unlimited. The ``BaseHeader`` value
for this attribute is ``None``; it is expected that specialized header
classes will override this value as needed.
``BaseHeader`` also provides the following method, which is called by the
email library code and should not in general be called by application
programs:
.. method:: fold(*, policy)
Return a string containing :attr:`~email.policy.Policy.linesep`
characters as required to correctly fold the header according to
*policy*. A :attr:`~email.policy.Policy.cte_type` of ``8bit`` will be
treated as if it were ``7bit``, since headers may not contain arbitrary
binary data. If :attr:`~email.policy.EmailPolicy.utf8` is ``False``,
non-ASCII data will be :rfc:`2047` encoded.
``BaseHeader`` by itself cannot be used to create a header object. It
defines a protocol that each specialized header cooperates with in order to
produce the header object. Specifically, ``BaseHeader`` requires that
the specialized class provide a :func:`classmethod` named ``parse``. This
method is called as follows::
parse(string, kwds)
``kwds`` is a dictionary containing one pre-initialized key, ``defects``.
``defects`` is an empty list. The parse method should append any detected
defects to this list. On return, the ``kwds`` dictionary *must* contain
values for at least the keys ``decoded`` and ``defects``. ``decoded``
should be the string value for the header (that is, the header value fully
decoded to unicode). The parse method should assume that *string* may
contain content-transfer-encoded parts, but should correctly handle all valid
unicode characters as well so that it can parse un-encoded header values.
``BaseHeader``'s ``__new__`` then creates the header instance, and calls its
``init`` method. The specialized class only needs to provide an ``init``
method if it wishes to set additional attributes beyond those provided by
``BaseHeader`` itself. Such an ``init`` method should look like this::
def init(self, *args, **kw):
self._myattr = kw.pop('myattr')
super().init(*args, **kw)
That is, anything extra that the specialized class puts in to the ``kwds``
dictionary should be removed and handled, and the remaining contents of
``kw`` (and ``args``) passed to the ``BaseHeader`` ``init`` method.
.. class:: UnstructuredHeader
An "unstructured" header is the default type of header in :rfc:`5322`.
Any header that does not have a specified syntax is treated as
unstructured. The classic example of an unstructured header is the
:mailheader:`Subject` header.
In :rfc:`5322`, an unstructured header is a run of arbitrary text in the
ASCII character set. :rfc:`2047`, however, has an :rfc:`5322` compatible
mechanism for encoding non-ASCII text as ASCII characters within a header
value. When a *value* containing encoded words is passed to the
constructor, the ``UnstructuredHeader`` parser converts such encoded words
into unicode, following the :rfc:`2047` rules for unstructured text. The
parser uses heuristics to attempt to decode certain non-compliant encoded
words. Defects are registered in such cases, as well as defects for issues
such as invalid characters within the encoded words or the non-encoded text.
This header type provides no additional attributes.
.. class:: DateHeader
:rfc:`5322` specifies a very specific format for dates within email headers.
The ``DateHeader`` parser recognizes that date format, as well as
recognizing a number of variant forms that are sometimes found "in the
wild".
This header type provides the following additional attributes:
.. attribute:: datetime
If the header value can be recognized as a valid date of one form or
another, this attribute will contain a :class:`~datetime.datetime`
instance representing that date. If the timezone of the input date is
specified as ``-0000`` (indicating it is in UTC but contains no
information about the source timezone), then :attr:`.datetime` will be a
naive :class:`~datetime.datetime`. If a specific timezone offset is
found (including `+0000`), then :attr:`.datetime` will contain an aware
``datetime`` that uses :class:`datetime.timezone` to record the timezone
offset.
The ``decoded`` value of the header is determined by formatting the
``datetime`` according to the :rfc:`5322` rules; that is, it is set to::
email.utils.format_datetime(self.datetime)
When creating a ``DateHeader``, *value* may be
:class:`~datetime.datetime` instance. This means, for example, that
the following code is valid and does what one would expect::
msg['Date'] = datetime(2011, 7, 15, 21)
Because this is a naive ``datetime`` it will be interpreted as a UTC
timestamp, and the resulting value will have a timezone of ``-0000``. Much
more useful is to use the :func:`~email.utils.localtime` function from the
:mod:`~email.utils` module::
msg['Date'] = utils.localtime()
This example sets the date header to the current time and date using
the current timezone offset.
.. class:: AddressHeader
Address headers are one of the most complex structured header types.
The ``AddressHeader`` class provides a generic interface to any address
header.
This header type provides the following additional attributes:
.. attribute:: groups
A tuple of :class:`.Group` objects encoding the
addresses and groups found in the header value. Addresses that are
not part of a group are represented in this list as single-address
``Groups`` whose :attr:`~.Group.display_name` is ``None``.
.. attribute:: addresses
A tuple of :class:`.Address` objects encoding all
of the individual addresses from the header value. If the header value
contains any groups, the individual addresses from the group are included
in the list at the point where the group occurs in the value (that is,
the list of addresses is "flattened" into a one dimensional list).
The ``decoded`` value of the header will have all encoded words decoded to
unicode. :class:`~encodings.idna` encoded domain names are also decoded to
unicode. The ``decoded`` value is set by :attr:`~str.join`\ ing the
:class:`str` value of the elements of the ``groups`` attribute with ``',
'``.
A list of :class:`.Address` and :class:`.Group` objects in any combination
may be used to set the value of an address header. ``Group`` objects whose
``display_name`` is ``None`` will be interpreted as single addresses, which
allows an address list to be copied with groups intact by using the list
obtained from the ``groups`` attribute of the source header.
.. class:: SingleAddressHeader
A subclass of :class:`.AddressHeader` that adds one
additional attribute:
.. attribute:: address
The single address encoded by the header value. If the header value
actually contains more than one address (which would be a violation of
the RFC under the default :mod:`~email.policy`), accessing this attribute
will result in a :exc:`ValueError`.
Many of the above classes also have a ``Unique`` variant (for example,
``UniqueUnstructuredHeader``). The only difference is that in the ``Unique``
variant, :attr:`~.BaseHeader.max_count` is set to 1.
.. class:: MIMEVersionHeader
There is really only one valid value for the :mailheader:`MIME-Version`
header, and that is ``1.0``. For future proofing, this header class
supports other valid version numbers. If a version number has a valid value
per :rfc:`2045`, then the header object will have non-``None`` values for
the following attributes:
.. attribute:: version
The version number as a string, with any whitespace and/or comments
removed.
.. attribute:: major
The major version number as an integer
.. attribute:: minor
The minor version number as an integer
.. class:: ParameterizedMIMEHeader
MIME headers all start with the prefix 'Content-'. Each specific header has
a certain value, described under the class for that header. Some can
also take a list of supplemental parameters, which have a common format.
This class serves as a base for all the MIME headers that take parameters.
.. attribute:: params
A dictionary mapping parameter names to parameter values.
.. class:: ContentTypeHeader
A :class:`ParameterizedMIMEHeader` class that handles the
:mailheader:`Content-Type` header.
.. attribute:: content_type
The content type string, in the form ``maintype/subtype``.
.. attribute:: maintype
.. attribute:: subtype
.. class:: ContentDispositionHeader
A :class:`ParameterizedMIMEHeader` class that handles the
:mailheader:`Content-Disposition` header.
.. attribute:: content-disposition
``inline`` and ``attachment`` are the only valid values in common use.
.. class:: ContentTransferEncoding
Handles the :mailheader:`Content-Transfer-Encoding` header.
.. attribute:: cte
Valid values are ``7bit``, ``8bit``, ``base64``, and
``quoted-printable``. See :rfc:`2045` for more information.
.. class:: HeaderRegistry(base_class=BaseHeader, \
default_class=UnstructuredHeader, \
use_default_map=True)
This is the factory used by :class:`~email.policy.EmailPolicy` by default.
``HeaderRegistry`` builds the class used to create a header instance
dynamically, using *base_class* and a specialized class retrieved from a
registry that it holds. When a given header name does not appear in the
registry, the class specified by *default_class* is used as the specialized
class. When *use_default_map* is ``True`` (the default), the standard
mapping of header names to classes is copied in to the registry during
initialization. *base_class* is always the last class in the generated
class's ``__bases__`` list.
The default mappings are:
:subject: UniqueUnstructuredHeader
:date: UniqueDateHeader
:resent-date: DateHeader
:orig-date: UniqueDateHeader
:sender: UniqueSingleAddressHeader
:resent-sender: SingleAddressHeader
:to: UniqueAddressHeader
:resent-to: AddressHeader
:cc: UniqueAddressHeader
:resent-cc: AddressHeader
:from: UniqueAddressHeader
:resent-from: AddressHeader
:reply-to: UniqueAddressHeader
``HeaderRegistry`` has the following methods:
.. method:: map_to_type(self, name, cls)
*name* is the name of the header to be mapped. It will be converted to
lower case in the registry. *cls* is the specialized class to be used,
along with *base_class*, to create the class used to instantiate headers
that match *name*.
.. method:: __getitem__(name)
Construct and return a class to handle creating a *name* header.
.. method:: __call__(name, value)
Retrieves the specialized header associated with *name* from the
registry (using *default_class* if *name* does not appear in the
registry) and composes it with *base_class* to produce a class,
calls the constructed class's constructor, passing it the same
argument list, and finally returns the class instance created thereby.
The following classes are the classes used to represent data parsed from
structured headers and can, in general, be used by an application program to
construct structured values to assign to specific headers.
.. class:: Address(display_name='', username='', domain='', addr_spec=None)
The class used to represent an email address. The general form of an
address is::
[display_name] <username@domain>
or::
username@domain
where each part must conform to specific syntax rules spelled out in
:rfc:`5322`.
As a convenience *addr_spec* can be specified instead of *username* and
*domain*, in which case *username* and *domain* will be parsed from the
*addr_spec*. An *addr_spec* must be a properly RFC quoted string; if it is
not ``Address`` will raise an error. Unicode characters are allowed and
will be property encoded when serialized. However, per the RFCs, unicode is
*not* allowed in the username portion of the address.
.. attribute:: display_name
The display name portion of the address, if any, with all quoting
removed. If the address does not have a display name, this attribute
will be an empty string.
.. attribute:: username
The ``username`` portion of the address, with all quoting removed.
.. attribute:: domain
The ``domain`` portion of the address.
.. attribute:: addr_spec
The ``username@domain`` portion of the address, correctly quoted
for use as a bare address (the second form shown above). This
attribute is not mutable.
.. method:: __str__()
The ``str`` value of the object is the address quoted according to
:rfc:`5322` rules, but with no Content Transfer Encoding of any non-ASCII
characters.
To support SMTP (:rfc:`5321`), ``Address`` handles one special case: if
``username`` and ``domain`` are both the empty string (or ``None``), then
the string value of the ``Address`` is ``<>``.
.. class:: Group(display_name=None, addresses=None)
The class used to represent an address group. The general form of an
address group is::
display_name: [address-list];
As a convenience for processing lists of addresses that consist of a mixture
of groups and single addresses, a ``Group`` may also be used to represent
single addresses that are not part of a group by setting *display_name* to
``None`` and providing a list of the single address as *addresses*.
.. attribute:: display_name
The ``display_name`` of the group. If it is ``None`` and there is
exactly one ``Address`` in ``addresses``, then the ``Group`` represents a
single address that is not in a group.
.. attribute:: addresses
A possibly empty tuple of :class:`.Address` objects representing the
addresses in the group.
.. method:: __str__()
The ``str`` value of a ``Group`` is formatted according to :rfc:`5322`,
but with no Content Transfer Encoding of any non-ASCII characters. If
``display_name`` is none and there is a single ``Address`` in the
``addresses`` list, the ``str`` value will be the same as the ``str`` of
that single ``Address``.
.. rubric:: Footnotes
.. [1] Originally added in 3.3 as a :term:`provisional module <provisional
package>`

83
third_party/python/Doc/library/email.iterators.rst vendored Normal file
View file

@ -0,0 +1,83 @@
:mod:`email.iterators`: Iterators
---------------------------------
.. module:: email.iterators
:synopsis: Iterate over a message object tree.
**Source code:** :source:`Lib/email/iterators.py`
--------------
Iterating over a message object tree is fairly easy with the
:meth:`Message.walk <email.message.Message.walk>` method. The
:mod:`email.iterators` module provides some useful higher level iterations over
message object trees.
.. function:: body_line_iterator(msg, decode=False)
This iterates over all the payloads in all the subparts of *msg*, returning the
string payloads line-by-line. It skips over all the subpart headers, and it
skips over any subpart with a payload that isn't a Python string. This is
somewhat equivalent to reading the flat text representation of the message from
a file using :meth:`~io.TextIOBase.readline`, skipping over all the
intervening headers.
Optional *decode* is passed through to :meth:`Message.get_payload
<email.message.Message.get_payload>`.
.. function:: typed_subpart_iterator(msg, maintype='text', subtype=None)
This iterates over all the subparts of *msg*, returning only those subparts that
match the MIME type specified by *maintype* and *subtype*.
Note that *subtype* is optional; if omitted, then subpart MIME type matching is
done only with the main type. *maintype* is optional too; it defaults to
:mimetype:`text`.
Thus, by default :func:`typed_subpart_iterator` returns each subpart that has a
MIME type of :mimetype:`text/\*`.
The following function has been added as a useful debugging tool. It should
*not* be considered part of the supported public interface for the package.
.. function:: _structure(msg, fp=None, level=0, include_default=False)
Prints an indented representation of the content types of the message object
structure. For example:
.. testsetup::
import email
from email.iterators import _structure
somefile = open('../Lib/test/test_email/data/msg_02.txt')
.. doctest::
>>> msg = email.message_from_file(somefile)
>>> _structure(msg)
multipart/mixed
text/plain
text/plain
multipart/digest
message/rfc822
text/plain
message/rfc822
text/plain
message/rfc822
text/plain
message/rfc822
text/plain
message/rfc822
text/plain
text/plain
.. testcleanup::
somefile.close()
Optional *fp* is a file-like object to print the output to. It must be
suitable for Python's :func:`print` function. *level* is used internally.
*include_default*, if true, prints the default type as well.

751
third_party/python/Doc/library/email.message.rst vendored Normal file
View file

@ -0,0 +1,751 @@
:mod:`email.message`: Representing an email message
---------------------------------------------------
.. module:: email.message
:synopsis: The base class representing email messages.
.. moduleauthor:: R. David Murray <rdmurray@bitdance.com>
.. sectionauthor:: R. David Murray <rdmurray@bitdance.com>,
Barry A. Warsaw <barry@python.org>
**Source code:** :source:`Lib/email/message.py`
--------------
.. versionadded:: 3.6 [1]_
The central class in the :mod:`email` package is the :class:`EmailMessage`
class, imported from the :mod:`email.message` module. It is the base class for
the :mod:`email` object model. :class:`EmailMessage` provides the core
functionality for setting and querying header fields, for accessing message
bodies, and for creating or modifying structured messages.
An email message consists of *headers* and a *payload* (which is also referred
to as the *content*). Headers are :rfc:`5322` or :rfc:`6532` style field names
and values, where the field name and value are separated by a colon. The colon
is not part of either the field name or the field value. The payload may be a
simple text message, or a binary object, or a structured sequence of
sub-messages each with their own set of headers and their own payload. The
latter type of payload is indicated by the message having a MIME type such as
:mimetype:`multipart/\*` or :mimetype:`message/rfc822`.
The conceptual model provided by an :class:`EmailMessage` object is that of an
ordered dictionary of headers coupled with a *payload* that represents the
:rfc:`5322` body of the message, which might be a list of sub-``EmailMessage``
objects. In addition to the normal dictionary methods for accessing the header
names and values, there are methods for accessing specialized information from
the headers (for example the MIME content type), for operating on the payload,
for generating a serialized version of the message, and for recursively walking
over the object tree.
The :class:`EmailMessage` dictionary-like interface is indexed by the header
names, which must be ASCII values. The values of the dictionary are strings
with some extra methods. Headers are stored and returned in case-preserving
form, but field names are matched case-insensitively. Unlike a real dict,
there is an ordering to the keys, and there can be duplicate keys. Additional
methods are provided for working with headers that have duplicate keys.
The *payload* is either a string or bytes object, in the case of simple message
objects, or a list of :class:`EmailMessage` objects, for MIME container
documents such as :mimetype:`multipart/\*` and :mimetype:`message/rfc822`
message objects.
.. class:: EmailMessage(policy=default)
If *policy* is specified use the rules it specifies to update and serialize
the representation of the message. If *policy* is not set, use the
:class:`~email.policy.default` policy, which follows the rules of the email
RFCs except for line endings (instead of the RFC mandated ``\r\n``, it uses
the Python standard ``\n`` line endings). For more information see the
:mod:`~email.policy` documentation.
.. method:: as_string(unixfrom=False, maxheaderlen=None, policy=None)
Return the entire message flattened as a string. When optional
*unixfrom* is true, the envelope header is included in the returned
string. *unixfrom* defaults to ``False``. For backward compatibility
with the base :class:`~email.message.Message` class *maxheaderlen* is
accepted, but defaults to ``None``, which means that by default the line
length is controlled by the
:attr:`~email.policy.EmailPolicy.max_line_length` of the policy. The
*policy* argument may be used to override the default policy obtained
from the message instance. This can be used to control some of the
formatting produced by the method, since the specified *policy* will be
passed to the :class:`~email.generator.Generator`.
Flattening the message may trigger changes to the :class:`EmailMessage`
if defaults need to be filled in to complete the transformation to a
string (for example, MIME boundaries may be generated or modified).
Note that this method is provided as a convenience and may not be the
most useful way to serialize messages in your application, especially if
you are dealing with multiple messages. See
:class:`email.generator.Generator` for a more flexible API for
serializing messages. Note also that this method is restricted to
producing messages serialized as "7 bit clean" when
:attr:`~email.policy.EmailPolicy.utf8` is ``False``, which is the default.
.. versionchanged:: 3.6 the default behavior when *maxheaderlen*
is not specified was changed from defaulting to 0 to defaulting
to the value of *max_line_length* from the policy.
.. method:: __str__()
Equivalent to ``as_string(policy=self.policy.clone(utf8=True))``. Allows
``str(msg)`` to produce a string containing the serialized message in a
readable format.
.. versionchanged:: 3.4 the method was changed to use ``utf8=True``,
thus producing an :rfc:`6531`-like message representation, instead of
being a direct alias for :meth:`as_string`.
.. method:: as_bytes(unixfrom=False, policy=None)
Return the entire message flattened as a bytes object. When optional
*unixfrom* is true, the envelope header is included in the returned
string. *unixfrom* defaults to ``False``. The *policy* argument may be
used to override the default policy obtained from the message instance.
This can be used to control some of the formatting produced by the
method, since the specified *policy* will be passed to the
:class:`~email.generator.BytesGenerator`.
Flattening the message may trigger changes to the :class:`EmailMessage`
if defaults need to be filled in to complete the transformation to a
string (for example, MIME boundaries may be generated or modified).
Note that this method is provided as a convenience and may not be the
most useful way to serialize messages in your application, especially if
you are dealing with multiple messages. See
:class:`email.generator.BytesGenerator` for a more flexible API for
serializing messages.
.. method:: __bytes__()
Equivalent to :meth:`.as_bytes()`. Allows ``bytes(msg)`` to produce a
bytes object containing the serialized message.
.. method:: is_multipart()
Return ``True`` if the message's payload is a list of
sub-\ :class:`EmailMessage` objects, otherwise return ``False``. When
:meth:`is_multipart` returns ``False``, the payload should be a string
object (which might be a CTE encoded binary payload). Note that
:meth:`is_multipart` returning ``True`` does not necessarily mean that
"msg.get_content_maintype() == 'multipart'" will return the ``True``.
For example, ``is_multipart`` will return ``True`` when the
:class:`EmailMessage` is of type ``message/rfc822``.
.. method:: set_unixfrom(unixfrom)
Set the message's envelope header to *unixfrom*, which should be a
string. (See :class:`~mailbox.mboxMessage` for a brief description of
this header.)
.. method:: get_unixfrom()
Return the message's envelope header. Defaults to ``None`` if the
envelope header was never set.
The following methods implement the mapping-like interface for accessing the
message's headers. Note that there are some semantic differences
between these methods and a normal mapping (i.e. dictionary) interface. For
example, in a dictionary there are no duplicate keys, but here there may be
duplicate message headers. Also, in dictionaries there is no guaranteed
order to the keys returned by :meth:`keys`, but in an :class:`EmailMessage`
object, headers are always returned in the order they appeared in the
original message, or in which they were added to the message later. Any
header deleted and then re-added is always appended to the end of the
header list.
These semantic differences are intentional and are biased toward
convenience in the most common use cases.
Note that in all cases, any envelope header present in the message is not
included in the mapping interface.
.. method:: __len__()
Return the total number of headers, including duplicates.
.. method:: __contains__(name)
Return true if the message object has a field named *name*. Matching is
done without regard to case and *name* does not include the trailing
colon. Used for the ``in`` operator. For example::
if 'message-id' in myMessage:
print('Message-ID:', myMessage['message-id'])
.. method:: __getitem__(name)
Return the value of the named header field. *name* does not include the
colon field separator. If the header is missing, ``None`` is returned; a
:exc:`KeyError` is never raised.
Note that if the named field appears more than once in the message's
headers, exactly which of those field values will be returned is
undefined. Use the :meth:`get_all` method to get the values of all the
extant headers named *name*.
Using the standard (non-``compat32``) policies, the returned value is an
instance of a subclass of :class:`email.headerregistry.BaseHeader`.
.. method:: __setitem__(name, val)
Add a header to the message with field name *name* and value *val*. The
field is appended to the end of the message's existing headers.
Note that this does *not* overwrite or delete any existing header with the same
name. If you want to ensure that the new header is the only one present in the
message with field name *name*, delete the field first, e.g.::
del msg['subject']
msg['subject'] = 'Python roolz!'
If the :mod:`policy` defines certain headers to be unique (as the standard
policies do), this method may raise a :exc:`ValueError` when an attempt
is made to assign a value to such a header when one already exists. This
behavior is intentional for consistency's sake, but do not depend on it
as we may choose to make such assignments do an automatic deletion of the
existing header in the future.
.. method:: __delitem__(name)
Delete all occurrences of the field with name *name* from the message's
headers. No exception is raised if the named field isn't present in the
headers.
.. method:: keys()
Return a list of all the message's header field names.
.. method:: values()
Return a list of all the message's field values.
.. method:: items()
Return a list of 2-tuples containing all the message's field headers and
values.
.. method:: get(name, failobj=None)
Return the value of the named header field. This is identical to
:meth:`__getitem__` except that optional *failobj* is returned if the
named header is missing (*failobj* defaults to ``None``).
Here are some additional useful header related methods:
.. method:: get_all(name, failobj=None)
Return a list of all the values for the field named *name*. If there are
no such named headers in the message, *failobj* is returned (defaults to
``None``).
.. method:: add_header(_name, _value, **_params)
Extended header setting. This method is similar to :meth:`__setitem__`
except that additional header parameters can be provided as keyword
arguments. *_name* is the header field to add and *_value* is the
*primary* value for the header.
For each item in the keyword argument dictionary *_params*, the key is
taken as the parameter name, with underscores converted to dashes (since
dashes are illegal in Python identifiers). Normally, the parameter will
be added as ``key="value"`` unless the value is ``None``, in which case
only the key will be added.
If the value contains non-ASCII characters, the charset and language may
be explicitly controlled by specifying the value as a three tuple in the
format ``(CHARSET, LANGUAGE, VALUE)``, where ``CHARSET`` is a string
naming the charset to be used to encode the value, ``LANGUAGE`` can
usually be set to ``None`` or the empty string (see :rfc:`2231` for other
possibilities), and ``VALUE`` is the string value containing non-ASCII
code points. If a three tuple is not passed and the value contains
non-ASCII characters, it is automatically encoded in :rfc:`2231` format
using a ``CHARSET`` of ``utf-8`` and a ``LANGUAGE`` of ``None``.
Here is an example::
msg.add_header('Content-Disposition', 'attachment', filename='bud.gif')
This will add a header that looks like ::
Content-Disposition: attachment; filename="bud.gif"
An example of the extended interface with non-ASCII characters::
msg.add_header('Content-Disposition', 'attachment',
filename=('iso-8859-1', '', 'Fußballer.ppt'))
.. method:: replace_header(_name, _value)
Replace a header. Replace the first header found in the message that
matches *_name*, retaining header order and field name case of the
original header. If no matching header is found, raise a
:exc:`KeyError`.
.. method:: get_content_type()
Return the message's content type, coerced to lower case of the form
:mimetype:`maintype/subtype`. If there is no :mailheader:`Content-Type`
header in the message return the value returned by
:meth:`get_default_type`. If the :mailheader:`Content-Type` header is
invalid, return ``text/plain``.
(According to :rfc:`2045`, messages always have a default type,
:meth:`get_content_type` will always return a value. :rfc:`2045` defines
a message's default type to be :mimetype:`text/plain` unless it appears
inside a :mimetype:`multipart/digest` container, in which case it would
be :mimetype:`message/rfc822`. If the :mailheader:`Content-Type` header
has an invalid type specification, :rfc:`2045` mandates that the default
type be :mimetype:`text/plain`.)
.. method:: get_content_maintype()
Return the message's main content type. This is the :mimetype:`maintype`
part of the string returned by :meth:`get_content_type`.
.. method:: get_content_subtype()
Return the message's sub-content type. This is the :mimetype:`subtype`
part of the string returned by :meth:`get_content_type`.
.. method:: get_default_type()
Return the default content type. Most messages have a default content
type of :mimetype:`text/plain`, except for messages that are subparts of
:mimetype:`multipart/digest` containers. Such subparts have a default
content type of :mimetype:`message/rfc822`.
.. method:: set_default_type(ctype)
Set the default content type. *ctype* should either be
:mimetype:`text/plain` or :mimetype:`message/rfc822`, although this is
not enforced. The default content type is not stored in the
:mailheader:`Content-Type` header, so it only affects the return value of
the ``get_content_type`` methods when no :mailheader:`Content-Type`
header is present in the message.
.. method:: set_param(param, value, header='Content-Type', requote=True, \
charset=None, language='', replace=False)
Set a parameter in the :mailheader:`Content-Type` header. If the
parameter already exists in the header, replace its value with *value*.
When *header* is ``Content-Type`` (the default) and the header does not
yet exist in the message, add it, set its value to
:mimetype:`text/plain`, and append the new parameter value. Optional
*header* specifies an alternative header to :mailheader:`Content-Type`.
If the value contains non-ASCII characters, the charset and language may
be explicitly specified using the optional *charset* and *language*
parameters. Optional *language* specifies the :rfc:`2231` language,
defaulting to the empty string. Both *charset* and *language* should be
strings. The default is to use the ``utf8`` *charset* and ``None`` for
the *language*.
If *replace* is ``False`` (the default) the header is moved to the
end of the list of headers. If *replace* is ``True``, the header
will be updated in place.
Use of the *requote* parameter with :class:`EmailMessage` objects is
deprecated.
Note that existing parameter values of headers may be accessed through
the :attr:`~email.headerregistry.BaseHeader.params` attribute of the
header value (for example, ``msg['Content-Type'].params['charset']``).
.. versionchanged:: 3.4 ``replace`` keyword was added.
.. method:: del_param(param, header='content-type', requote=True)
Remove the given parameter completely from the :mailheader:`Content-Type`
header. The header will be re-written in place without the parameter or
its value. Optional *header* specifies an alternative to
:mailheader:`Content-Type`.
Use of the *requote* parameter with :class:`EmailMessage` objects is
deprecated.
.. method:: get_filename(failobj=None)
Return the value of the ``filename`` parameter of the
:mailheader:`Content-Disposition` header of the message. If the header
does not have a ``filename`` parameter, this method falls back to looking
for the ``name`` parameter on the :mailheader:`Content-Type` header. If
neither is found, or the header is missing, then *failobj* is returned.
The returned string will always be unquoted as per
:func:`email.utils.unquote`.
.. method:: get_boundary(failobj=None)
Return the value of the ``boundary`` parameter of the
:mailheader:`Content-Type` header of the message, or *failobj* if either
the header is missing, or has no ``boundary`` parameter. The returned
string will always be unquoted as per :func:`email.utils.unquote`.
.. method:: set_boundary(boundary)
Set the ``boundary`` parameter of the :mailheader:`Content-Type` header to
*boundary*. :meth:`set_boundary` will always quote *boundary* if
necessary. A :exc:`~email.errors.HeaderParseError` is raised if the
message object has no :mailheader:`Content-Type` header.
Note that using this method is subtly different from deleting the old
:mailheader:`Content-Type` header and adding a new one with the new
boundary via :meth:`add_header`, because :meth:`set_boundary` preserves
the order of the :mailheader:`Content-Type` header in the list of
headers.
.. method:: get_content_charset(failobj=None)
Return the ``charset`` parameter of the :mailheader:`Content-Type` header,
coerced to lower case. If there is no :mailheader:`Content-Type` header, or if
that header has no ``charset`` parameter, *failobj* is returned.
.. method:: get_charsets(failobj=None)
Return a list containing the character set names in the message. If the
message is a :mimetype:`multipart`, then the list will contain one element
for each subpart in the payload, otherwise, it will be a list of length 1.
Each item in the list will be a string which is the value of the
``charset`` parameter in the :mailheader:`Content-Type` header for the
represented subpart. If the subpart has no :mailheader:`Content-Type`
header, no ``charset`` parameter, or is not of the :mimetype:`text` main
MIME type, then that item in the returned list will be *failobj*.
.. method:: is_attachment
Return ``True`` if there is a :mailheader:`Content-Disposition` header
and its (case insensitive) value is ``attachment``, ``False`` otherwise.
.. versionchanged:: 3.4.2
is_attachment is now a method instead of a property, for consistency
with :meth:`~email.message.Message.is_multipart`.
.. method:: get_content_disposition()
Return the lowercased value (without parameters) of the message's
:mailheader:`Content-Disposition` header if it has one, or ``None``. The
possible values for this method are *inline*, *attachment* or ``None``
if the message follows :rfc:`2183`.
.. versionadded:: 3.5
The following methods relate to interrogating and manipulating the content
(payload) of the message.
.. method:: walk()
The :meth:`walk` method is an all-purpose generator which can be used to
iterate over all the parts and subparts of a message object tree, in
depth-first traversal order. You will typically use :meth:`walk` as the
iterator in a ``for`` loop; each iteration returns the next subpart.
Here's an example that prints the MIME type of every part of a multipart
message structure:
.. testsetup::
from email import message_from_binary_file
with open('../Lib/test/test_email/data/msg_16.txt', 'rb') as f:
msg = message_from_binary_file(f)
from email.iterators import _structure
.. doctest::
>>> for part in msg.walk():
... print(part.get_content_type())
multipart/report
text/plain
message/delivery-status
text/plain
text/plain
message/rfc822
text/plain
``walk`` iterates over the subparts of any part where
:meth:`is_multipart` returns ``True``, even though
``msg.get_content_maintype() == 'multipart'`` may return ``False``. We
can see this in our example by making use of the ``_structure`` debug
helper function:
.. doctest::
>>> for part in msg.walk():
... print(part.get_content_maintype() == 'multipart',
... part.is_multipart())
True True
False False
False True
False False
False False
False True
False False
>>> _structure(msg)
multipart/report
text/plain
message/delivery-status
text/plain
text/plain
message/rfc822
text/plain
Here the ``message`` parts are not ``multiparts``, but they do contain
subparts. ``is_multipart()`` returns ``True`` and ``walk`` descends
into the subparts.
.. method:: get_body(preferencelist=('related', 'html', 'plain'))
Return the MIME part that is the best candidate to be the "body" of the
message.
*preferencelist* must be a sequence of strings from the set ``related``,
``html``, and ``plain``, and indicates the order of preference for the
content type of the part returned.
Start looking for candidate matches with the object on which the
``get_body`` method is called.
If ``related`` is not included in *preferencelist*, consider the root
part (or subpart of the root part) of any related encountered as a
candidate if the (sub-)part matches a preference.
When encountering a ``multipart/related``, check the ``start`` parameter
and if a part with a matching :mailheader:`Content-ID` is found, consider
only it when looking for candidate matches. Otherwise consider only the
first (default root) part of the ``multipart/related``.
If a part has a :mailheader:`Content-Disposition` header, only consider
the part a candidate match if the value of the header is ``inline``.
If none of the candidates matches any of the preferences in
*preferencelist*, return ``None``.
Notes: (1) For most applications the only *preferencelist* combinations
that really make sense are ``('plain',)``, ``('html', 'plain')``, and the
default ``('related', 'html', 'plain')``. (2) Because matching starts
with the object on which ``get_body`` is called, calling ``get_body`` on
a ``multipart/related`` will return the object itself unless
*preferencelist* has a non-default value. (3) Messages (or message parts)
that do not specify a :mailheader:`Content-Type` or whose
:mailheader:`Content-Type` header is invalid will be treated as if they
are of type ``text/plain``, which may occasionally cause ``get_body`` to
return unexpected results.
.. method:: iter_attachments()
Return an iterator over all of the immediate sub-parts of the message
that are not candidate "body" parts. That is, skip the first occurrence
of each of ``text/plain``, ``text/html``, ``multipart/related``, or
``multipart/alternative`` (unless they are explicitly marked as
attachments via :mailheader:`Content-Disposition: attachment`), and
return all remaining parts. When applied directly to a
``multipart/related``, return an iterator over the all the related parts
except the root part (ie: the part pointed to by the ``start`` parameter,
or the first part if there is no ``start`` parameter or the ``start``
parameter doesn't match the :mailheader:`Content-ID` of any of the
parts). When applied directly to a ``multipart/alternative`` or a
non-``multipart``, return an empty iterator.
.. method:: iter_parts()
Return an iterator over all of the immediate sub-parts of the message,
which will be empty for a non-``multipart``. (See also
:meth:`~email.message.EmailMessage.walk`.)
.. method:: get_content(*args, content_manager=None, **kw)
Call the :meth:`~email.contentmanager.ContentManager.get_content` method
of the *content_manager*, passing self as the message object, and passing
along any other arguments or keywords as additional arguments. If
*content_manager* is not specified, use the ``content_manager`` specified
by the current :mod:`~email.policy`.
.. method:: set_content(*args, content_manager=None, **kw)
Call the :meth:`~email.contentmanager.ContentManager.set_content` method
of the *content_manager*, passing self as the message object, and passing
along any other arguments or keywords as additional arguments. If
*content_manager* is not specified, use the ``content_manager`` specified
by the current :mod:`~email.policy`.
.. method:: make_related(boundary=None)
Convert a non-``multipart`` message into a ``multipart/related`` message,
moving any existing :mailheader:`Content-` headers and payload into a
(new) first part of the ``multipart``. If *boundary* is specified, use
it as the boundary string in the multipart, otherwise leave the boundary
to be automatically created when it is needed (for example, when the
message is serialized).
.. method:: make_alternative(boundary=None)
Convert a non-``multipart`` or a ``multipart/related`` into a
``multipart/alternative``, moving any existing :mailheader:`Content-`
headers and payload into a (new) first part of the ``multipart``. If
*boundary* is specified, use it as the boundary string in the multipart,
otherwise leave the boundary to be automatically created when it is
needed (for example, when the message is serialized).
.. method:: make_mixed(boundary=None)
Convert a non-``multipart``, a ``multipart/related``, or a
``multipart-alternative`` into a ``multipart/mixed``, moving any existing
:mailheader:`Content-` headers and payload into a (new) first part of the
``multipart``. If *boundary* is specified, use it as the boundary string
in the multipart, otherwise leave the boundary to be automatically
created when it is needed (for example, when the message is serialized).
.. method:: add_related(*args, content_manager=None, **kw)
If the message is a ``multipart/related``, create a new message
object, pass all of the arguments to its :meth:`set_content` method,
and :meth:`~email.message.Message.attach` it to the ``multipart``. If
the message is a non-``multipart``, call :meth:`make_related` and then
proceed as above. If the message is any other type of ``multipart``,
raise a :exc:`TypeError`. If *content_manager* is not specified, use
the ``content_manager`` specified by the current :mod:`~email.policy`.
If the added part has no :mailheader:`Content-Disposition` header,
add one with the value ``inline``.
.. method:: add_alternative(*args, content_manager=None, **kw)
If the message is a ``multipart/alternative``, create a new message
object, pass all of the arguments to its :meth:`set_content` method, and
:meth:`~email.message.Message.attach` it to the ``multipart``. If the
message is a non-``multipart`` or ``multipart/related``, call
:meth:`make_alternative` and then proceed as above. If the message is
any other type of ``multipart``, raise a :exc:`TypeError`. If
*content_manager* is not specified, use the ``content_manager`` specified
by the current :mod:`~email.policy`.
.. method:: add_attachment(*args, content_manager=None, **kw)
If the message is a ``multipart/mixed``, create a new message object,
pass all of the arguments to its :meth:`set_content` method, and
:meth:`~email.message.Message.attach` it to the ``multipart``. If the
message is a non-``multipart``, ``multipart/related``, or
``multipart/alternative``, call :meth:`make_mixed` and then proceed as
above. If *content_manager* is not specified, use the ``content_manager``
specified by the current :mod:`~email.policy`. If the added part
has no :mailheader:`Content-Disposition` header, add one with the value
``attachment``. This method can be used both for explicit attachments
(:mailheader:`Content-Disposition: attachment`) and ``inline`` attachments
(:mailheader:`Content-Disposition: inline`), by passing appropriate
options to the ``content_manager``.
.. method:: clear()
Remove the payload and all of the headers.
.. method:: clear_content()
Remove the payload and all of the :exc:`Content-` headers, leaving
all other headers intact and in their original order.
:class:`EmailMessage` objects have the following instance attributes:
.. attribute:: preamble
The format of a MIME document allows for some text between the blank line
following the headers, and the first multipart boundary string. Normally,
this text is never visible in a MIME-aware mail reader because it falls
outside the standard MIME armor. However, when viewing the raw text of
the message, or when viewing the message in a non-MIME aware reader, this
text can become visible.
The *preamble* attribute contains this leading extra-armor text for MIME
documents. When the :class:`~email.parser.Parser` discovers some text
after the headers but before the first boundary string, it assigns this
text to the message's *preamble* attribute. When the
:class:`~email.generator.Generator` is writing out the plain text
representation of a MIME message, and it finds the
message has a *preamble* attribute, it will write this text in the area
between the headers and the first boundary. See :mod:`email.parser` and
:mod:`email.generator` for details.
Note that if the message object has no preamble, the *preamble* attribute
will be ``None``.
.. attribute:: epilogue
The *epilogue* attribute acts the same way as the *preamble* attribute,
except that it contains text that appears between the last boundary and
the end of the message. As with the :attr:`~EmailMessage.preamble`,
if there is no epilog text this attribute will be ``None``.
.. attribute:: defects
The *defects* attribute contains a list of all the problems found when
parsing this message. See :mod:`email.errors` for a detailed description
of the possible parsing defects.
.. class:: MIMEPart(policy=default)
This class represents a subpart of a MIME message. It is identical to
:class:`EmailMessage`, except that no :mailheader:`MIME-Version` headers are
added when :meth:`~EmailMessage.set_content` is called, since sub-parts do
not need their own :mailheader:`MIME-Version` headers.
.. rubric:: Footnotes
.. [1] Originally added in 3.4 as a :term:`provisional module <provisional
package>`. Docs for legacy message class moved to
:ref:`compat32_message`.

259
third_party/python/Doc/library/email.mime.rst vendored Normal file
View file

@ -0,0 +1,259 @@
:mod:`email.mime`: Creating email and MIME objects from scratch
---------------------------------------------------------------
.. module:: email.mime
:synopsis: Build MIME messages.
**Source code:** :source:`Lib/email/mime/`
--------------
This module is part of the legacy (``Compat32``) email API. Its functionality
is partially replaced by the :mod:`~email.contentmanager` in the new API, but
in certain applications these classes may still be useful, even in non-legacy
code.
Ordinarily, you get a message object structure by passing a file or some text to
a parser, which parses the text and returns the root message object. However
you can also build a complete message structure from scratch, or even individual
:class:`~email.message.Message` objects by hand. In fact, you can also take an
existing structure and add new :class:`~email.message.Message` objects, move them
around, etc. This makes a very convenient interface for slicing-and-dicing MIME
messages.
You can create a new object structure by creating :class:`~email.message.Message`
instances, adding attachments and all the appropriate headers manually. For MIME
messages though, the :mod:`email` package provides some convenient subclasses to
make things easier.
Here are the classes:
.. currentmodule:: email.mime.base
.. class:: MIMEBase(_maintype, _subtype, *, policy=compat32, **_params)
Module: :mod:`email.mime.base`
This is the base class for all the MIME-specific subclasses of
:class:`~email.message.Message`. Ordinarily you won't create instances
specifically of :class:`MIMEBase`, although you could. :class:`MIMEBase`
is provided primarily as a convenient base class for more specific
MIME-aware subclasses.
*_maintype* is the :mailheader:`Content-Type` major type (e.g. :mimetype:`text`
or :mimetype:`image`), and *_subtype* is the :mailheader:`Content-Type` minor
type (e.g. :mimetype:`plain` or :mimetype:`gif`). *_params* is a parameter
key/value dictionary and is passed directly to :meth:`Message.add_header
<email.message.Message.add_header>`.
If *policy* is specified, (defaults to the
:class:`compat32 <email.policy.Compat32>` policy) it will be passed to
:class:`~email.message.Message`.
The :class:`MIMEBase` class always adds a :mailheader:`Content-Type` header
(based on *_maintype*, *_subtype*, and *_params*), and a
:mailheader:`MIME-Version` header (always set to ``1.0``).
.. versionchanged:: 3.6
Added *policy* keyword-only parameter.
.. currentmodule:: email.mime.nonmultipart
.. class:: MIMENonMultipart()
Module: :mod:`email.mime.nonmultipart`
A subclass of :class:`~email.mime.base.MIMEBase`, this is an intermediate base
class for MIME messages that are not :mimetype:`multipart`. The primary
purpose of this class is to prevent the use of the
:meth:`~email.message.Message.attach` method, which only makes sense for
:mimetype:`multipart` messages. If :meth:`~email.message.Message.attach`
is called, a :exc:`~email.errors.MultipartConversionError` exception is raised.
.. currentmodule:: email.mime.multipart
.. class:: MIMEMultipart(_subtype='mixed', boundary=None, _subparts=None, \
*, policy=compat32, **_params)
Module: :mod:`email.mime.multipart`
A subclass of :class:`~email.mime.base.MIMEBase`, this is an intermediate base
class for MIME messages that are :mimetype:`multipart`. Optional *_subtype*
defaults to :mimetype:`mixed`, but can be used to specify the subtype of the
message. A :mailheader:`Content-Type` header of :mimetype:`multipart/_subtype`
will be added to the message object. A :mailheader:`MIME-Version` header will
also be added.
Optional *boundary* is the multipart boundary string. When ``None`` (the
default), the boundary is calculated when needed (for example, when the
message is serialized).
*_subparts* is a sequence of initial subparts for the payload. It must be
possible to convert this sequence to a list. You can always attach new subparts
to the message by using the :meth:`Message.attach
<email.message.Message.attach>` method.
Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`.
Additional parameters for the :mailheader:`Content-Type` header are taken from
the keyword arguments, or passed into the *_params* argument, which is a keyword
dictionary.
.. versionchanged:: 3.6
Added *policy* keyword-only parameter.
.. currentmodule:: email.mime.application
.. class:: MIMEApplication(_data, _subtype='octet-stream', \
_encoder=email.encoders.encode_base64, \
*, policy=compat32, **_params)
Module: :mod:`email.mime.application`
A subclass of :class:`~email.mime.nonmultipart.MIMENonMultipart`, the
:class:`MIMEApplication` class is used to represent MIME message objects of
major type :mimetype:`application`. *_data* is a string containing the raw
byte data. Optional *_subtype* specifies the MIME subtype and defaults to
:mimetype:`octet-stream`.
Optional *_encoder* is a callable (i.e. function) which will perform the actual
encoding of the data for transport. This callable takes one argument, which is
the :class:`MIMEApplication` instance. It should use
:meth:`~email.message.Message.get_payload` and
:meth:`~email.message.Message.set_payload` to change the payload to encoded
form. It should also add
any :mailheader:`Content-Transfer-Encoding` or other headers to the message
object as necessary. The default encoding is base64. See the
:mod:`email.encoders` module for a list of the built-in encoders.
Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`.
*_params* are passed straight through to the base class constructor.
.. versionchanged:: 3.6
Added *policy* keyword-only parameter.
.. currentmodule:: email.mime.audio
.. class:: MIMEAudio(_audiodata, _subtype=None, \
_encoder=email.encoders.encode_base64, \
*, policy=compat32, **_params)
Module: :mod:`email.mime.audio`
A subclass of :class:`~email.mime.nonmultipart.MIMENonMultipart`, the
:class:`MIMEAudio` class is used to create MIME message objects of major type
:mimetype:`audio`. *_audiodata* is a string containing the raw audio data. If
this data can be decoded by the standard Python module :mod:`sndhdr`, then the
subtype will be automatically included in the :mailheader:`Content-Type` header.
Otherwise you can explicitly specify the audio subtype via the *_subtype*
argument. If the minor type could not be guessed and *_subtype* was not given,
then :exc:`TypeError` is raised.
Optional *_encoder* is a callable (i.e. function) which will perform the actual
encoding of the audio data for transport. This callable takes one argument,
which is the :class:`MIMEAudio` instance. It should use
:meth:`~email.message.Message.get_payload` and
:meth:`~email.message.Message.set_payload` to change the payload to encoded
form. It should also add
any :mailheader:`Content-Transfer-Encoding` or other headers to the message
object as necessary. The default encoding is base64. See the
:mod:`email.encoders` module for a list of the built-in encoders.
Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`.
*_params* are passed straight through to the base class constructor.
.. versionchanged:: 3.6
Added *policy* keyword-only parameter.
.. currentmodule:: email.mime.image
.. class:: MIMEImage(_imagedata, _subtype=None, \
_encoder=email.encoders.encode_base64, \
*, policy=compat32, **_params)
Module: :mod:`email.mime.image`
A subclass of :class:`~email.mime.nonmultipart.MIMENonMultipart`, the
:class:`MIMEImage` class is used to create MIME message objects of major type
:mimetype:`image`. *_imagedata* is a string containing the raw image data. If
this data can be decoded by the standard Python module :mod:`imghdr`, then the
subtype will be automatically included in the :mailheader:`Content-Type` header.
Otherwise you can explicitly specify the image subtype via the *_subtype*
argument. If the minor type could not be guessed and *_subtype* was not given,
then :exc:`TypeError` is raised.
Optional *_encoder* is a callable (i.e. function) which will perform the actual
encoding of the image data for transport. This callable takes one argument,
which is the :class:`MIMEImage` instance. It should use
:meth:`~email.message.Message.get_payload` and
:meth:`~email.message.Message.set_payload` to change the payload to encoded
form. It should also add
any :mailheader:`Content-Transfer-Encoding` or other headers to the message
object as necessary. The default encoding is base64. See the
:mod:`email.encoders` module for a list of the built-in encoders.
Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`.
*_params* are passed straight through to the :class:`~email.mime.base.MIMEBase`
constructor.
.. versionchanged:: 3.6
Added *policy* keyword-only parameter.
.. currentmodule:: email.mime.message
.. class:: MIMEMessage(_msg, _subtype='rfc822', *, policy=compat32)
Module: :mod:`email.mime.message`
A subclass of :class:`~email.mime.nonmultipart.MIMENonMultipart`, the
:class:`MIMEMessage` class is used to create MIME objects of main type
:mimetype:`message`. *_msg* is used as the payload, and must be an instance
of class :class:`~email.message.Message` (or a subclass thereof), otherwise
a :exc:`TypeError` is raised.
Optional *_subtype* sets the subtype of the message; it defaults to
:mimetype:`rfc822`.
Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`.
.. versionchanged:: 3.6
Added *policy* keyword-only parameter.
.. currentmodule:: email.mime.text
.. class:: MIMEText(_text, _subtype='plain', _charset=None, *, policy=compat32)
Module: :mod:`email.mime.text`
A subclass of :class:`~email.mime.nonmultipart.MIMENonMultipart`, the
:class:`MIMEText` class is used to create MIME objects of major type
:mimetype:`text`. *_text* is the string for the payload. *_subtype* is the
minor type and defaults to :mimetype:`plain`. *_charset* is the character
set of the text and is passed as an argument to the
:class:`~email.mime.nonmultipart.MIMENonMultipart` constructor; it defaults
to ``us-ascii`` if the string contains only ``ascii`` code points, and
``utf-8`` otherwise. The *_charset* parameter accepts either a string or a
:class:`~email.charset.Charset` instance.
Unless the *_charset* argument is explicitly set to ``None``, the
MIMEText object created will have both a :mailheader:`Content-Type` header
with a ``charset`` parameter, and a :mailheader:`Content-Transfer-Encoding`
header. This means that a subsequent ``set_payload`` call will not result
in an encoded payload, even if a charset is passed in the ``set_payload``
command. You can "reset" this behavior by deleting the
``Content-Transfer-Encoding`` header, after which a ``set_payload`` call
will automatically encode the new payload (and add a new
:mailheader:`Content-Transfer-Encoding` header).
Optional *policy* argument defaults to :class:`compat32 <email.policy.Compat32>`.
.. versionchanged:: 3.5
*_charset* also accepts :class:`~email.charset.Charset` instances.
.. versionchanged:: 3.6
Added *policy* keyword-only parameter.

320
third_party/python/Doc/library/email.parser.rst vendored Normal file
View file

@ -0,0 +1,320 @@
:mod:`email.parser`: Parsing email messages
-------------------------------------------
.. module:: email.parser
:synopsis: Parse flat text email messages to produce a message object structure.
**Source code:** :source:`Lib/email/parser.py`
--------------
Message object structures can be created in one of two ways: they can be
created from whole cloth by creating an :class:`~email.message.EmailMessage`
object, adding headers using the dictionary interface, and adding payload(s)
using :meth:`~email.message.EmailMessage.set_content` and related methods, or
they can be created by parsing a serialized representation of the email
message.
The :mod:`email` package provides a standard parser that understands most email
document structures, including MIME documents. You can pass the parser a
bytes, string or file object, and the parser will return to you the root
:class:`~email.message.EmailMessage` instance of the object structure. For
simple, non-MIME messages the payload of this root object will likely be a
string containing the text of the message. For MIME messages, the root object
will return ``True`` from its :meth:`~email.message.EmailMessage.is_multipart`
method, and the subparts can be accessed via the payload manipulation methods,
such as :meth:`~email.message.EmailMessage.get_body`,
:meth:`~email.message.EmailMessage.iter_parts`, and
:meth:`~email.message.EmailMessage.walk`.
There are actually two parser interfaces available for use, the :class:`Parser`
API and the incremental :class:`FeedParser` API. The :class:`Parser` API is
most useful if you have the entire text of the message in memory, or if the
entire message lives in a file on the file system. :class:`FeedParser` is more
appropriate when you are reading the message from a stream which might block
waiting for more input (such as reading an email message from a socket). The
:class:`FeedParser` can consume and parse the message incrementally, and only
returns the root object when you close the parser.
Note that the parser can be extended in limited ways, and of course you can
implement your own parser completely from scratch. All of the logic that
connects the :mod:`email` package's bundled parser and the
:class:`~email.message.EmailMessage` class is embodied in the :mod:`policy`
class, so a custom parser can create message object trees any way it finds
necessary by implementing custom versions of the appropriate :mod:`policy`
methods.
FeedParser API
^^^^^^^^^^^^^^
The :class:`BytesFeedParser`, imported from the :mod:`email.feedparser` module,
provides an API that is conducive to incremental parsing of email messages,
such as would be necessary when reading the text of an email message from a
source that can block (such as a socket). The :class:`BytesFeedParser` can of
course be used to parse an email message fully contained in a :term:`bytes-like
object`, string, or file, but the :class:`BytesParser` API may be more
convenient for such use cases. The semantics and results of the two parser
APIs are identical.
The :class:`BytesFeedParser`'s API is simple; you create an instance, feed it a
bunch of bytes until there's no more to feed it, then close the parser to
retrieve the root message object. The :class:`BytesFeedParser` is extremely
accurate when parsing standards-compliant messages, and it does a very good job
of parsing non-compliant messages, providing information about how a message
was deemed broken. It will populate a message object's
:attr:`~email.message.EmailMessage.defects` attribute with a list of any
problems it found in a message. See the :mod:`email.errors` module for the
list of defects that it can find.
Here is the API for the :class:`BytesFeedParser`:
.. class:: BytesFeedParser(_factory=None, *, policy=policy.compat32)
Create a :class:`BytesFeedParser` instance. Optional *_factory* is a
no-argument callable; if not specified use the
:attr:`~email.policy.Policy.message_factory` from the *policy*. Call
*_factory* whenever a new message object is needed.
If *policy* is specified use the rules it specifies to update the
representation of the message. If *policy* is not set, use the
:class:`compat32 <email.policy.Compat32>` policy, which maintains backward
compatibility with the Python 3.2 version of the email package and provides
:class:`~email.message.Message` as the default factory. All other policies
provide :class:`~email.message.EmailMessage` as the default *_factory*. For
more information on what else *policy* controls, see the
:mod:`~email.policy` documentation.
Note: **The policy keyword should always be specified**; The default will
change to :data:`email.policy.default` in a future version of Python.
.. versionadded:: 3.2
.. versionchanged:: 3.3 Added the *policy* keyword.
.. versionchanged:: 3.6 *_factory* defaults to the policy ``message_factory``.
.. method:: feed(data)
Feed the parser some more data. *data* should be a :term:`bytes-like
object` containing one or more lines. The lines can be partial and the
parser will stitch such partial lines together properly. The lines can
have any of the three common line endings: carriage return, newline, or
carriage return and newline (they can even be mixed).
.. method:: close()
Complete the parsing of all previously fed data and return the root
message object. It is undefined what happens if :meth:`~feed` is called
after this method has been called.
.. class:: FeedParser(_factory=None, *, policy=policy.compat32)
Works like :class:`BytesFeedParser` except that the input to the
:meth:`~BytesFeedParser.feed` method must be a string. This is of limited
utility, since the only way for such a message to be valid is for it to
contain only ASCII text or, if :attr:`~email.policy.Policy.utf8` is
``True``, no binary attachments.
.. versionchanged:: 3.3 Added the *policy* keyword.
Parser API
^^^^^^^^^^
The :class:`BytesParser` class, imported from the :mod:`email.parser` module,
provides an API that can be used to parse a message when the complete contents
of the message are available in a :term:`bytes-like object` or file. The
:mod:`email.parser` module also provides :class:`Parser` for parsing strings,
and header-only parsers, :class:`BytesHeaderParser` and
:class:`HeaderParser`, which can be used if you're only interested in the
headers of the message. :class:`BytesHeaderParser` and :class:`HeaderParser`
can be much faster in these situations, since they do not attempt to parse the
message body, instead setting the payload to the raw body.
.. class:: BytesParser(_class=None, *, policy=policy.compat32)
Create a :class:`BytesParser` instance. The *_class* and *policy*
arguments have the same meaning and semantics as the *_factory*
and *policy* arguments of :class:`BytesFeedParser`.
Note: **The policy keyword should always be specified**; The default will
change to :data:`email.policy.default` in a future version of Python.
.. versionchanged:: 3.3
Removed the *strict* argument that was deprecated in 2.4. Added the
*policy* keyword.
.. versionchanged:: 3.6 *_class* defaults to the policy ``message_factory``.
.. method:: parse(fp, headersonly=False)
Read all the data from the binary file-like object *fp*, parse the
resulting bytes, and return the message object. *fp* must support
both the :meth:`~io.IOBase.readline` and the :meth:`~io.IOBase.read`
methods.
The bytes contained in *fp* must be formatted as a block of :rfc:`5322`
(or, if :attr:`~email.policy.Policy.utf8` is ``True``, :rfc:`6532`)
style headers and header continuation lines, optionally preceded by an
envelope header. The header block is terminated either by the end of the
data or by a blank line. Following the header block is the body of the
message (which may contain MIME-encoded subparts, including subparts
with a :mailheader:`Content-Transfer-Encoding` of ``8bit``).
Optional *headersonly* is a flag specifying whether to stop parsing after
reading the headers or not. The default is ``False``, meaning it parses
the entire contents of the file.
.. method:: parsebytes(bytes, headersonly=False)
Similar to the :meth:`parse` method, except it takes a :term:`bytes-like
object` instead of a file-like object. Calling this method on a
:term:`bytes-like object` is equivalent to wrapping *bytes* in a
:class:`~io.BytesIO` instance first and calling :meth:`parse`.
Optional *headersonly* is as with the :meth:`parse` method.
.. versionadded:: 3.2
.. class:: BytesHeaderParser(_class=None, *, policy=policy.compat32)
Exactly like :class:`BytesParser`, except that *headersonly*
defaults to ``True``.
.. versionadded:: 3.3
.. class:: Parser(_class=None, *, policy=policy.compat32)
This class is parallel to :class:`BytesParser`, but handles string input.
.. versionchanged:: 3.3
Removed the *strict* argument. Added the *policy* keyword.
.. versionchanged:: 3.6 *_class* defaults to the policy ``message_factory``.
.. method:: parse(fp, headersonly=False)
Read all the data from the text-mode file-like object *fp*, parse the
resulting text, and return the root message object. *fp* must support
both the :meth:`~io.TextIOBase.readline` and the
:meth:`~io.TextIOBase.read` methods on file-like objects.
Other than the text mode requirement, this method operates like
:meth:`BytesParser.parse`.
.. method:: parsestr(text, headersonly=False)
Similar to the :meth:`parse` method, except it takes a string object
instead of a file-like object. Calling this method on a string is
equivalent to wrapping *text* in a :class:`~io.StringIO` instance first
and calling :meth:`parse`.
Optional *headersonly* is as with the :meth:`parse` method.
.. class:: HeaderParser(_class=None, *, policy=policy.compat32)
Exactly like :class:`Parser`, except that *headersonly*
defaults to ``True``.
Since creating a message object structure from a string or a file object is such
a common task, four functions are provided as a convenience. They are available
in the top-level :mod:`email` package namespace.
.. currentmodule:: email
.. function:: message_from_bytes(s, _class=None, *, policy=policy.compat32)
Return a message object structure from a :term:`bytes-like object`. This is
equivalent to ``BytesParser().parsebytes(s)``. Optional *_class* and
*policy* are interpreted as with the :class:`~email.parser.BytesParser` class
constructor.
.. versionadded:: 3.2
.. versionchanged:: 3.3
Removed the *strict* argument. Added the *policy* keyword.
.. function:: message_from_binary_file(fp, _class=None, *, \
policy=policy.compat32)
Return a message object structure tree from an open binary :term:`file
object`. This is equivalent to ``BytesParser().parse(fp)``. *_class* and
*policy* are interpreted as with the :class:`~email.parser.BytesParser` class
constructor.
.. versionadded:: 3.2
.. versionchanged:: 3.3
Removed the *strict* argument. Added the *policy* keyword.
.. function:: message_from_string(s, _class=None, *, policy=policy.compat32)
Return a message object structure from a string. This is equivalent to
``Parser().parsestr(s)``. *_class* and *policy* are interpreted as
with the :class:`~email.parser.Parser` class constructor.
.. versionchanged:: 3.3
Removed the *strict* argument. Added the *policy* keyword.
.. function:: message_from_file(fp, _class=None, *, policy=policy.compat32)
Return a message object structure tree from an open :term:`file object`.
This is equivalent to ``Parser().parse(fp)``. *_class* and *policy* are
interpreted as with the :class:`~email.parser.Parser` class constructor.
.. versionchanged:: 3.3
Removed the *strict* argument. Added the *policy* keyword.
.. versionchanged:: 3.6 *_class* defaults to the policy ``message_factory``.
Here's an example of how you might use :func:`message_from_bytes` at an
interactive Python prompt::
>>> import email
>>> msg = email.message_from_bytes(myBytes) # doctest: +SKIP
Additional notes
^^^^^^^^^^^^^^^^
Here are some notes on the parsing semantics:
* Most non-\ :mimetype:`multipart` type messages are parsed as a single message
object with a string payload. These objects will return ``False`` for
:meth:`~email.message.EmailMessage.is_multipart`, and
:meth:`~email.message.EmailMessage.iter_parts` will yield an empty list.
* All :mimetype:`multipart` type messages will be parsed as a container message
object with a list of sub-message objects for their payload. The outer
container message will return ``True`` for
:meth:`~email.message.EmailMessage.is_multipart`, and
:meth:`~email.message.EmailMessage.iter_parts` will yield a list of subparts.
* Most messages with a content type of :mimetype:`message/\*` (such as
:mimetype:`message/delivery-status` and :mimetype:`message/rfc822`) will also
be parsed as container object containing a list payload of length 1. Their
:meth:`~email.message.EmailMessage.is_multipart` method will return ``True``.
The single element yielded by :meth:`~email.message.EmailMessage.iter_parts`
will be a sub-message object.
* Some non-standards-compliant messages may not be internally consistent about
their :mimetype:`multipart`\ -edness. Such messages may have a
:mailheader:`Content-Type` header of type :mimetype:`multipart`, but their
:meth:`~email.message.EmailMessage.is_multipart` method may return ``False``.
If such messages were parsed with the :class:`~email.parser.FeedParser`,
they will have an instance of the
:class:`~email.errors.MultipartInvariantViolationDefect` class in their
*defects* attribute list. See :mod:`email.errors` for details.

651
third_party/python/Doc/library/email.policy.rst vendored Normal file
View file

@ -0,0 +1,651 @@
:mod:`email.policy`: Policy Objects
-----------------------------------
.. module:: email.policy
:synopsis: Controlling the parsing and generating of messages
.. moduleauthor:: R. David Murray <rdmurray@bitdance.com>
.. sectionauthor:: R. David Murray <rdmurray@bitdance.com>
.. versionadded:: 3.3
**Source code:** :source:`Lib/email/policy.py`
--------------
The :mod:`email` package's prime focus is the handling of email messages as
described by the various email and MIME RFCs. However, the general format of
email messages (a block of header fields each consisting of a name followed by
a colon followed by a value, the whole block followed by a blank line and an
arbitrary 'body'), is a format that has found utility outside of the realm of
email. Some of these uses conform fairly closely to the main email RFCs, some
do not. Even when working with email, there are times when it is desirable to
break strict compliance with the RFCs, such as generating emails that
interoperate with email servers that do not themselves follow the standards, or
that implement extensions you want to use in ways that violate the
standards.
Policy objects give the email package the flexibility to handle all these
disparate use cases.
A :class:`Policy` object encapsulates a set of attributes and methods that
control the behavior of various components of the email package during use.
:class:`Policy` instances can be passed to various classes and methods in the
email package to alter the default behavior. The settable values and their
defaults are described below.
There is a default policy used by all classes in the email package. For all of
the :mod:`~email.parser` classes and the related convenience functions, and for
the :class:`~email.message.Message` class, this is the :class:`Compat32`
policy, via its corresponding pre-defined instance :const:`compat32`. This
policy provides for complete backward compatibility (in some cases, including
bug compatibility) with the pre-Python3.3 version of the email package.
This default value for the *policy* keyword to
:class:`~email.message.EmailMessage` is the :class:`EmailPolicy` policy, via
its pre-defined instance :data:`~default`.
When a :class:`~email.message.Message` or :class:`~email.message.EmailMessage`
object is created, it acquires a policy. If the message is created by a
:mod:`~email.parser`, a policy passed to the parser will be the policy used by
the message it creates. If the message is created by the program, then the
policy can be specified when it is created. When a message is passed to a
:mod:`~email.generator`, the generator uses the policy from the message by
default, but you can also pass a specific policy to the generator that will
override the one stored on the message object.
The default value for the *policy* keyword for the :mod:`email.parser` classes
and the parser convenience functions **will be changing** in a future version of
Python. Therefore you should **always specify explicitly which policy you want
to use** when calling any of the classes and functions described in the
:mod:`~email.parser` module.
The first part of this documentation covers the features of :class:`Policy`, an
:term:`abstract base class` that defines the features that are common to all
policy objects, including :const:`compat32`. This includes certain hook
methods that are called internally by the email package, which a custom policy
could override to obtain different behavior. The second part describes the
concrete classes :class:`EmailPolicy` and :class:`Compat32`, which implement
the hooks that provide the standard behavior and the backward compatible
behavior and features, respectively.
:class:`Policy` instances are immutable, but they can be cloned, accepting the
same keyword arguments as the class constructor and returning a new
:class:`Policy` instance that is a copy of the original but with the specified
attributes values changed.
As an example, the following code could be used to read an email message from a
file on disk and pass it to the system ``sendmail`` program on a Unix system:
.. testsetup::
from unittest import mock
mocker = mock.patch('subprocess.Popen')
m = mocker.start()
proc = mock.MagicMock()
m.return_value = proc
proc.stdin.close.return_value = None
mymsg = open('mymsg.txt', 'w')
mymsg.write('To: abc@xyz.com\n\n')
mymsg.flush()
.. doctest::
>>> from email import message_from_binary_file
>>> from email.generator import BytesGenerator
>>> from email import policy
>>> from subprocess import Popen, PIPE
>>> with open('mymsg.txt', 'rb') as f:
... msg = message_from_binary_file(f, policy=policy.default)
>>> p = Popen(['sendmail', msg['To'].addresses[0]], stdin=PIPE)
>>> g = BytesGenerator(p.stdin, policy=msg.policy.clone(linesep='\r\n'))
>>> g.flatten(msg)
>>> p.stdin.close()
>>> rc = p.wait()
.. testcleanup::
mymsg.close()
mocker.stop()
import os
os.remove('mymsg.txt')
Here we are telling :class:`~email.generator.BytesGenerator` to use the RFC
correct line separator characters when creating the binary string to feed into
``sendmail's`` ``stdin``, where the default policy would use ``\n`` line
separators.
Some email package methods accept a *policy* keyword argument, allowing the
policy to be overridden for that method. For example, the following code uses
the :meth:`~email.message.Message.as_bytes` method of the *msg* object from
the previous example and writes the message to a file using the native line
separators for the platform on which it is running::
>>> import os
>>> with open('converted.txt', 'wb') as f:
... f.write(msg.as_bytes(policy=msg.policy.clone(linesep=os.linesep)))
17
Policy objects can also be combined using the addition operator, producing a
policy object whose settings are a combination of the non-default values of the
summed objects::
>>> compat_SMTP = policy.compat32.clone(linesep='\r\n')
>>> compat_strict = policy.compat32.clone(raise_on_defect=True)
>>> compat_strict_SMTP = compat_SMTP + compat_strict
This operation is not commutative; that is, the order in which the objects are
added matters. To illustrate::
>>> policy100 = policy.compat32.clone(max_line_length=100)
>>> policy80 = policy.compat32.clone(max_line_length=80)
>>> apolicy = policy100 + policy80
>>> apolicy.max_line_length
80
>>> apolicy = policy80 + policy100
>>> apolicy.max_line_length
100
.. class:: Policy(**kw)
This is the :term:`abstract base class` for all policy classes. It provides
default implementations for a couple of trivial methods, as well as the
implementation of the immutability property, the :meth:`clone` method, and
the constructor semantics.
The constructor of a policy class can be passed various keyword arguments.
The arguments that may be specified are any non-method properties on this
class, plus any additional non-method properties on the concrete class. A
value specified in the constructor will override the default value for the
corresponding attribute.
This class defines the following properties, and thus values for the
following may be passed in the constructor of any policy class:
.. attribute:: max_line_length
The maximum length of any line in the serialized output, not counting the
end of line character(s). Default is 78, per :rfc:`5322`. A value of
``0`` or :const:`None` indicates that no line wrapping should be
done at all.
.. attribute:: linesep
The string to be used to terminate lines in serialized output. The
default is ``\n`` because that's the internal end-of-line discipline used
by Python, though ``\r\n`` is required by the RFCs.
.. attribute:: cte_type
Controls the type of Content Transfer Encodings that may be or are
required to be used. The possible values are:
.. tabularcolumns:: |l|L|
======== ===============================================================
``7bit`` all data must be "7 bit clean" (ASCII-only). This means that
where necessary data will be encoded using either
quoted-printable or base64 encoding.
``8bit`` data is not constrained to be 7 bit clean. Data in headers is
still required to be ASCII-only and so will be encoded (see
:meth:`fold_binary` and :attr:`~EmailPolicy.utf8` below for
exceptions), but body parts may use the ``8bit`` CTE.
======== ===============================================================
A ``cte_type`` value of ``8bit`` only works with ``BytesGenerator``, not
``Generator``, because strings cannot contain binary data. If a
``Generator`` is operating under a policy that specifies
``cte_type=8bit``, it will act as if ``cte_type`` is ``7bit``.
.. attribute:: raise_on_defect
If :const:`True`, any defects encountered will be raised as errors. If
:const:`False` (the default), defects will be passed to the
:meth:`register_defect` method.
.. attribute:: mangle_from\_
If :const:`True`, lines starting with *"From "* in the body are
escaped by putting a ``>`` in front of them. This parameter is used when
the message is being serialized by a generator.
Default: :const:`False`.
.. versionadded:: 3.5
The *mangle_from_* parameter.
.. attribute:: message_factory
A factory function for constructing a new empty message object. Used
by the parser when building messages. Defaults to ``None``, in
which case :class:`~email.message.Message` is used.
.. versionadded:: 3.6
The following :class:`Policy` method is intended to be called by code using
the email library to create policy instances with custom settings:
.. method:: clone(**kw)
Return a new :class:`Policy` instance whose attributes have the same
values as the current instance, except where those attributes are
given new values by the keyword arguments.
The remaining :class:`Policy` methods are called by the email package code,
and are not intended to be called by an application using the email package.
A custom policy must implement all of these methods.
.. method:: handle_defect(obj, defect)
Handle a *defect* found on *obj*. When the email package calls this
method, *defect* will always be a subclass of
:class:`~email.errors.Defect`.
The default implementation checks the :attr:`raise_on_defect` flag. If
it is ``True``, *defect* is raised as an exception. If it is ``False``
(the default), *obj* and *defect* are passed to :meth:`register_defect`.
.. method:: register_defect(obj, defect)
Register a *defect* on *obj*. In the email package, *defect* will always
be a subclass of :class:`~email.errors.Defect`.
The default implementation calls the ``append`` method of the ``defects``
attribute of *obj*. When the email package calls :attr:`handle_defect`,
*obj* will normally have a ``defects`` attribute that has an ``append``
method. Custom object types used with the email package (for example,
custom ``Message`` objects) should also provide such an attribute,
otherwise defects in parsed messages will raise unexpected errors.
.. method:: header_max_count(name)
Return the maximum allowed number of headers named *name*.
Called when a header is added to an :class:`~email.message.EmailMessage`
or :class:`~email.message.Message` object. If the returned value is not
``0`` or ``None``, and there are already a number of headers with the
name *name* greater than or equal to the value returned, a
:exc:`ValueError` is raised.
Because the default behavior of ``Message.__setitem__`` is to append the
value to the list of headers, it is easy to create duplicate headers
without realizing it. This method allows certain headers to be limited
in the number of instances of that header that may be added to a
``Message`` programmatically. (The limit is not observed by the parser,
which will faithfully produce as many headers as exist in the message
being parsed.)
The default implementation returns ``None`` for all header names.
.. method:: header_source_parse(sourcelines)
The email package calls this method with a list of strings, each string
ending with the line separation characters found in the source being
parsed. The first line includes the field header name and separator.
All whitespace in the source is preserved. The method should return the
``(name, value)`` tuple that is to be stored in the ``Message`` to
represent the parsed header.
If an implementation wishes to retain compatibility with the existing
email package policies, *name* should be the case preserved name (all
characters up to the '``:``' separator), while *value* should be the
unfolded value (all line separator characters removed, but whitespace
kept intact), stripped of leading whitespace.
*sourcelines* may contain surrogateescaped binary data.
There is no default implementation
.. method:: header_store_parse(name, value)
The email package calls this method with the name and value provided by
the application program when the application program is modifying a
``Message`` programmatically (as opposed to a ``Message`` created by a
parser). The method should return the ``(name, value)`` tuple that is to
be stored in the ``Message`` to represent the header.
If an implementation wishes to retain compatibility with the existing
email package policies, the *name* and *value* should be strings or
string subclasses that do not change the content of the passed in
arguments.
There is no default implementation
.. method:: header_fetch_parse(name, value)
The email package calls this method with the *name* and *value* currently
stored in the ``Message`` when that header is requested by the
application program, and whatever the method returns is what is passed
back to the application as the value of the header being retrieved.
Note that there may be more than one header with the same name stored in
the ``Message``; the method is passed the specific name and value of the
header destined to be returned to the application.
*value* may contain surrogateescaped binary data. There should be no
surrogateescaped binary data in the value returned by the method.
There is no default implementation
.. method:: fold(name, value)
The email package calls this method with the *name* and *value* currently
stored in the ``Message`` for a given header. The method should return a
string that represents that header "folded" correctly (according to the
policy settings) by composing the *name* with the *value* and inserting
:attr:`linesep` characters at the appropriate places. See :rfc:`5322`
for a discussion of the rules for folding email headers.
*value* may contain surrogateescaped binary data. There should be no
surrogateescaped binary data in the string returned by the method.
.. method:: fold_binary(name, value)
The same as :meth:`fold`, except that the returned value should be a
bytes object rather than a string.
*value* may contain surrogateescaped binary data. These could be
converted back into binary data in the returned bytes object.
.. class:: EmailPolicy(**kw)
This concrete :class:`Policy` provides behavior that is intended to be fully
compliant with the current email RFCs. These include (but are not limited
to) :rfc:`5322`, :rfc:`2047`, and the current MIME RFCs.
This policy adds new header parsing and folding algorithms. Instead of
simple strings, headers are ``str`` subclasses with attributes that depend
on the type of the field. The parsing and folding algorithm fully implement
:rfc:`2047` and :rfc:`5322`.
The default value for the :attr:`~email.policy.Policy.message_factory`
attribute is :class:`~email.message.EmailMessage`.
In addition to the settable attributes listed above that apply to all
policies, this policy adds the following additional attributes:
.. versionadded:: 3.6 [1]_
.. attribute:: utf8
If ``False``, follow :rfc:`5322`, supporting non-ASCII characters in
headers by encoding them as "encoded words". If ``True``, follow
:rfc:`6532` and use ``utf-8`` encoding for headers. Messages
formatted in this way may be passed to SMTP servers that support
the ``SMTPUTF8`` extension (:rfc:`6531`).
.. attribute:: refold_source
If the value for a header in the ``Message`` object originated from a
:mod:`~email.parser` (as opposed to being set by a program), this
attribute indicates whether or not a generator should refold that value
when transforming the message back into serialized form. The possible
values are:
======== ===============================================================
``none`` all source values use original folding
``long`` source values that have any line that is longer than
``max_line_length`` will be refolded
``all`` all values are refolded.
======== ===============================================================
The default is ``long``.
.. attribute:: header_factory
A callable that takes two arguments, ``name`` and ``value``, where
``name`` is a header field name and ``value`` is an unfolded header field
value, and returns a string subclass that represents that header. A
default ``header_factory`` (see :mod:`~email.headerregistry`) is provided
that supports custom parsing for the various address and date :RFC:`5322`
header field types, and the major MIME header field stypes. Support for
additional custom parsing will be added in the future.
.. attribute:: content_manager
An object with at least two methods: get_content and set_content. When
the :meth:`~email.message.EmailMessage.get_content` or
:meth:`~email.message.EmailMessage.set_content` method of an
:class:`~email.message.EmailMessage` object is called, it calls the
corresponding method of this object, passing it the message object as its
first argument, and any arguments or keywords that were passed to it as
additional arguments. By default ``content_manager`` is set to
:data:`~email.contentmanager.raw_data_manager`.
.. versionadded:: 3.4
The class provides the following concrete implementations of the abstract
methods of :class:`Policy`:
.. method:: header_max_count(name)
Returns the value of the
:attr:`~email.headerregistry.BaseHeader.max_count` attribute of the
specialized class used to represent the header with the given name.
.. method:: header_source_parse(sourcelines)
The name is parsed as everything up to the '``:``' and returned
unmodified. The value is determined by stripping leading whitespace off
the remainder of the first line, joining all subsequent lines together,
and stripping any trailing carriage return or linefeed characters.
.. method:: header_store_parse(name, value)
The name is returned unchanged. If the input value has a ``name``
attribute and it matches *name* ignoring case, the value is returned
unchanged. Otherwise the *name* and *value* are passed to
``header_factory``, and the resulting header object is returned as
the value. In this case a ``ValueError`` is raised if the input value
contains CR or LF characters.
.. method:: header_fetch_parse(name, value)
If the value has a ``name`` attribute, it is returned to unmodified.
Otherwise the *name*, and the *value* with any CR or LF characters
removed, are passed to the ``header_factory``, and the resulting
header object is returned. Any surrogateescaped bytes get turned into
the unicode unknown-character glyph.
.. method:: fold(name, value)
Header folding is controlled by the :attr:`refold_source` policy setting.
A value is considered to be a 'source value' if and only if it does not
have a ``name`` attribute (having a ``name`` attribute means it is a
header object of some sort). If a source value needs to be refolded
according to the policy, it is converted into a header object by
passing the *name* and the *value* with any CR and LF characters removed
to the ``header_factory``. Folding of a header object is done by
calling its ``fold`` method with the current policy.
Source values are split into lines using :meth:`~str.splitlines`. If
the value is not to be refolded, the lines are rejoined using the
``linesep`` from the policy and returned. The exception is lines
containing non-ascii binary data. In that case the value is refolded
regardless of the ``refold_source`` setting, which causes the binary data
to be CTE encoded using the ``unknown-8bit`` charset.
.. method:: fold_binary(name, value)
The same as :meth:`fold` if :attr:`~Policy.cte_type` is ``7bit``, except
that the returned value is bytes.
If :attr:`~Policy.cte_type` is ``8bit``, non-ASCII binary data is
converted back
into bytes. Headers with binary data are not refolded, regardless of the
``refold_header`` setting, since there is no way to know whether the
binary data consists of single byte characters or multibyte characters.
The following instances of :class:`EmailPolicy` provide defaults suitable for
specific application domains. Note that in the future the behavior of these
instances (in particular the ``HTTP`` instance) may be adjusted to conform even
more closely to the RFCs relevant to their domains.
.. data:: default
An instance of ``EmailPolicy`` with all defaults unchanged. This policy
uses the standard Python ``\n`` line endings rather than the RFC-correct
``\r\n``.
.. data:: SMTP
Suitable for serializing messages in conformance with the email RFCs.
Like ``default``, but with ``linesep`` set to ``\r\n``, which is RFC
compliant.
.. data:: SMTPUTF8
The same as ``SMTP`` except that :attr:`~EmailPolicy.utf8` is ``True``.
Useful for serializing messages to a message store without using encoded
words in the headers. Should only be used for SMTP transmission if the
sender or recipient addresses have non-ASCII characters (the
:meth:`smtplib.SMTP.send_message` method handles this automatically).
.. data:: HTTP
Suitable for serializing headers with for use in HTTP traffic. Like
``SMTP`` except that ``max_line_length`` is set to ``None`` (unlimited).
.. data:: strict
Convenience instance. The same as ``default`` except that
``raise_on_defect`` is set to ``True``. This allows any policy to be made
strict by writing::
somepolicy + policy.strict
With all of these :class:`EmailPolicies <.EmailPolicy>`, the effective API of
the email package is changed from the Python 3.2 API in the following ways:
* Setting a header on a :class:`~email.message.Message` results in that
header being parsed and a header object created.
* Fetching a header value from a :class:`~email.message.Message` results
in that header being parsed and a header object created and
returned.
* Any header object, or any header that is refolded due to the
policy settings, is folded using an algorithm that fully implements the
RFC folding algorithms, including knowing where encoded words are required
and allowed.
From the application view, this means that any header obtained through the
:class:`~email.message.EmailMessage` is a header object with extra
attributes, whose string value is the fully decoded unicode value of the
header. Likewise, a header may be assigned a new value, or a new header
created, using a unicode string, and the policy will take care of converting
the unicode string into the correct RFC encoded form.
The header objects and their attributes are described in
:mod:`~email.headerregistry`.
.. class:: Compat32(**kw)
This concrete :class:`Policy` is the backward compatibility policy. It
replicates the behavior of the email package in Python 3.2. The
:mod:`~email.policy` module also defines an instance of this class,
:const:`compat32`, that is used as the default policy. Thus the default
behavior of the email package is to maintain compatibility with Python 3.2.
The following attributes have values that are different from the
:class:`Policy` default:
.. attribute:: mangle_from_
The default is ``True``.
The class provides the following concrete implementations of the
abstract methods of :class:`Policy`:
.. method:: header_source_parse(sourcelines)
The name is parsed as everything up to the '``:``' and returned
unmodified. The value is determined by stripping leading whitespace off
the remainder of the first line, joining all subsequent lines together,
and stripping any trailing carriage return or linefeed characters.
.. method:: header_store_parse(name, value)
The name and value are returned unmodified.
.. method:: header_fetch_parse(name, value)
If the value contains binary data, it is converted into a
:class:`~email.header.Header` object using the ``unknown-8bit`` charset.
Otherwise it is returned unmodified.
.. method:: fold(name, value)
Headers are folded using the :class:`~email.header.Header` folding
algorithm, which preserves existing line breaks in the value, and wraps
each resulting line to the ``max_line_length``. Non-ASCII binary data are
CTE encoded using the ``unknown-8bit`` charset.
.. method:: fold_binary(name, value)
Headers are folded using the :class:`~email.header.Header` folding
algorithm, which preserves existing line breaks in the value, and wraps
each resulting line to the ``max_line_length``. If ``cte_type`` is
``7bit``, non-ascii binary data is CTE encoded using the ``unknown-8bit``
charset. Otherwise the original source header is used, with its existing
line breaks and any (RFC invalid) binary data it may contain.
.. data:: compat32
An instance of :class:`Compat32`, providing backward compatibility with the
behavior of the email package in Python 3.2.
.. rubric:: Footnotes
.. [1] Originally added in 3.3 as a :term:`provisional feature <provisional
package>`.

152
third_party/python/Doc/library/email.rst vendored Normal file
View file

@ -0,0 +1,152 @@
:mod:`email` --- An email and MIME handling package
===================================================
.. module:: email
:synopsis: Package supporting the parsing, manipulating, and generating
email messages.
.. moduleauthor:: Barry A. Warsaw <barry@python.org>,
R. David Murray <rdmurray@bitdance.com>
.. sectionauthor:: R. David Murray <rdmurray@bitdance.com>
**Source code:** :source:`Lib/email/__init__.py`
--------------
The :mod:`email` package is a library for managing email messages. It is
specifically *not* designed to do any sending of email messages to SMTP
(:rfc:`2821`), NNTP, or other servers; those are functions of modules such as
:mod:`smtplib` and :mod:`nntplib`. The :mod:`email` package attempts to be as
RFC-compliant as possible, supporting :rfc:`5233` and :rfc:`6532`, as well as
such MIME-related RFCs as :rfc:`2045`, :rfc:`2046`, :rfc:`2047`, :rfc:`2183`,
and :rfc:`2231`.
The overall structure of the email package can be divided into three major
components, plus a fourth component that controls the behavior of the other
components.
The central component of the package is an "object model" that represents email
messages. An application interacts with the package primarily through the
object model interface defined in the :mod:`~email.message` sub-module. The
application can use this API to ask questions about an existing email, to
construct a new email, or to add or remove email subcomponents that themselves
use the same object model interface. That is, following the nature of email
messages and their MIME subcomponents, the email object model is a tree
structure of objects that all provide the :class:`~email.message.EmailMessage`
API.
The other two major components of the package are the :mod:`~email.parser` and
the :mod:`~email.generator`. The parser takes the serialized version of an
email message (a stream of bytes) and converts it into a tree of
:class:`~email.message.EmailMessage` objects. The generator takes an
:class:`~email.message.EmailMessage` and turns it back into a serialized byte
stream. (The parser and generator also handle streams of text characters, but
this usage is discouraged as it is too easy to end up with messages that are
not valid in one way or another.)
The control component is the :mod:`~email.policy` module. Every
:class:`~email.message.EmailMessage`, every :mod:`~email.generator`, and every
:mod:`~email.parser` has an associated :mod:`~email.policy` object that
controls its behavior. Usually an application only needs to specify the policy
when an :class:`~email.message.EmailMessage` is created, either by directly
instantiating an :class:`~email.message.EmailMessage` to create a new email,
or by parsing an input stream using a :mod:`~email.parser`. But the policy can
be changed when the message is serialized using a :mod:`~email.generator`.
This allows, for example, a generic email message to be parsed from disk, but
to serialize it using standard SMTP settings when sending it to an email
server.
The email package does its best to hide the details of the various governing
RFCs from the application. Conceptually the application should be able to
treat the email message as a structured tree of unicode text and binary
attachments, without having to worry about how these are represented when
serialized. In practice, however, it is often necessary to be aware of at
least some of the rules governing MIME messages and their structure,
specifically the names and nature of the MIME "content types" and how they
identify multipart documents. For the most part this knowledge should only be
required for more complex applications, and even then it should only be the
high level structure in question, and not the details of how those structures
are represented. Since MIME content types are used widely in modern internet
software (not just email), this will be a familiar concept to many programmers.
The following sections describe the functionality of the :mod:`email` package.
We start with the :mod:`~email.message` object model, which is the primary
interface an application will use, and follow that with the
:mod:`~email.parser` and :mod:`~email.generator` components. Then we cover the
:mod:`~email.policy` controls, which completes the treatment of the main
components of the library.
The next three sections cover the exceptions the package may raise and the
defects (non-compliance with the RFCs) that the :mod:`~email.parser` may
detect. Then we cover the :mod:`~email.headerregistry` and the
:mod:`~email.contentmanager` sub-components, which provide tools for doing more
detailed manipulation of headers and payloads, respectively. Both of these
components contain features relevant to consuming and producing non-trivial
messages, but also document their extensibility APIs, which will be of interest
to advanced applications.
Following those is a set of examples of using the fundamental parts of the APIs
covered in the preceding sections.
The foregoing represent the modern (unicode friendly) API of the email package.
The remaining sections, starting with the :class:`~email.message.Message`
class, cover the legacy :data:`~email.policy.compat32` API that deals much more
directly with the details of how email messages are represented. The
:data:`~email.policy.compat32` API does *not* hide the details of the RFCs from
the application, but for applications that need to operate at that level, they
can be useful tools. This documentation is also relevant for applications that
are still using the :mod:`~email.policy.compat32` API for backward
compatibility reasons.
.. versionchanged:: 3.6
Docs reorganized and rewritten to promote the new
:class:`~email.message.EmailMessage`/:class:`~email.policy.EmailPolicy`
API.
Contents of the :mod:`email` package documentation:
.. toctree::
email.message.rst
email.parser.rst
email.generator.rst
email.policy.rst
email.errors.rst
email.headerregistry.rst
email.contentmanager.rst
email.examples.rst
Legacy API:
.. toctree::
email.compat32-message.rst
email.mime.rst
email.header.rst
email.charset.rst
email.encoders.rst
email.utils.rst
email.iterators.rst
.. seealso::
Module :mod:`smtplib`
SMTP (Simple Mail Transport Protocol) client
Module :mod:`poplib`
POP (Post Office Protocol) client
Module :mod:`imaplib`
IMAP (Internet Message Access Protocol) client
Module :mod:`nntplib`
NNTP (Net News Transport Protocol) client
Module :mod:`mailbox`
Tools for creating, reading, and managing collections of messages on disk
using a variety standard formats.
Module :mod:`smtpd`
SMTP server framework (primarily useful for testing)

218
third_party/python/Doc/library/email.utils.rst vendored Normal file
View file

@ -0,0 +1,218 @@
:mod:`email.utils`: Miscellaneous utilities
-------------------------------------------
.. module:: email.utils
:synopsis: Miscellaneous email package utilities.
**Source code:** :source:`Lib/email/utils.py`
--------------
There are a couple of useful utilities provided in the :mod:`email.utils`
module:
.. function:: localtime(dt=None)
Return local time as an aware datetime object. If called without
arguments, return current time. Otherwise *dt* argument should be a
:class:`~datetime.datetime` instance, and it is converted to the local time
zone according to the system time zone database. If *dt* is naive (that
is, ``dt.tzinfo`` is ``None``), it is assumed to be in local time. In this
case, a positive or zero value for *isdst* causes ``localtime`` to presume
initially that summer time (for example, Daylight Saving Time) is or is not
(respectively) in effect for the specified time. A negative value for
*isdst* causes the ``localtime`` to attempt to divine whether summer time
is in effect for the specified time.
.. versionadded:: 3.3
.. function:: make_msgid(idstring=None, domain=None)
Returns a string suitable for an :rfc:`2822`\ -compliant
:mailheader:`Message-ID` header. Optional *idstring* if given, is a string
used to strengthen the uniqueness of the message id. Optional *domain* if
given provides the portion of the msgid after the '@'. The default is the
local hostname. It is not normally necessary to override this default, but
may be useful certain cases, such as a constructing distributed system that
uses a consistent domain name across multiple hosts.
.. versionchanged:: 3.2
Added the *domain* keyword.
The remaining functions are part of the legacy (``Compat32``) email API. There
is no need to directly use these with the new API, since the parsing and
formatting they provide is done automatically by the header parsing machinery
of the new API.
.. function:: quote(str)
Return a new string with backslashes in *str* replaced by two backslashes, and
double quotes replaced by backslash-double quote.
.. function:: unquote(str)
Return a new string which is an *unquoted* version of *str*. If *str* ends and
begins with double quotes, they are stripped off. Likewise if *str* ends and
begins with angle brackets, they are stripped off.
.. function:: parseaddr(address)
Parse address -- which should be the value of some address-containing field such
as :mailheader:`To` or :mailheader:`Cc` -- into its constituent *realname* and
*email address* parts. Returns a tuple of that information, unless the parse
fails, in which case a 2-tuple of ``('', '')`` is returned.
.. function:: formataddr(pair, charset='utf-8')
The inverse of :meth:`parseaddr`, this takes a 2-tuple of the form ``(realname,
email_address)`` and returns the string value suitable for a :mailheader:`To` or
:mailheader:`Cc` header. If the first element of *pair* is false, then the
second element is returned unmodified.
Optional *charset* is the character set that will be used in the :rfc:`2047`
encoding of the ``realname`` if the ``realname`` contains non-ASCII
characters. Can be an instance of :class:`str` or a
:class:`~email.charset.Charset`. Defaults to ``utf-8``.
.. versionchanged:: 3.3
Added the *charset* option.
.. function:: getaddresses(fieldvalues)
This method returns a list of 2-tuples of the form returned by ``parseaddr()``.
*fieldvalues* is a sequence of header field values as might be returned by
:meth:`Message.get_all <email.message.Message.get_all>`. Here's a simple
example that gets all the recipients of a message::
from email.utils import getaddresses
tos = msg.get_all('to', [])
ccs = msg.get_all('cc', [])
resent_tos = msg.get_all('resent-to', [])
resent_ccs = msg.get_all('resent-cc', [])
all_recipients = getaddresses(tos + ccs + resent_tos + resent_ccs)
.. function:: parsedate(date)
Attempts to parse a date according to the rules in :rfc:`2822`. however, some
mailers don't follow that format as specified, so :func:`parsedate` tries to
guess correctly in such cases. *date* is a string containing an :rfc:`2822`
date, such as ``"Mon, 20 Nov 1995 19:12:08 -0500"``. If it succeeds in parsing
the date, :func:`parsedate` returns a 9-tuple that can be passed directly to
:func:`time.mktime`; otherwise ``None`` will be returned. Note that indexes 6,
7, and 8 of the result tuple are not usable.
.. function:: parsedate_tz(date)
Performs the same function as :func:`parsedate`, but returns either ``None`` or
a 10-tuple; the first 9 elements make up a tuple that can be passed directly to
:func:`time.mktime`, and the tenth is the offset of the date's timezone from UTC
(which is the official term for Greenwich Mean Time) [#]_. If the input string
has no timezone, the last element of the tuple returned is ``None``. Note that
indexes 6, 7, and 8 of the result tuple are not usable.
.. function:: parsedate_to_datetime(date)
The inverse of :func:`format_datetime`. Performs the same function as
:func:`parsedate`, but on success returns a :mod:`~datetime.datetime`. If
the input date has a timezone of ``-0000``, the ``datetime`` will be a naive
``datetime``, and if the date is conforming to the RFCs it will represent a
time in UTC but with no indication of the actual source timezone of the
message the date comes from. If the input date has any other valid timezone
offset, the ``datetime`` will be an aware ``datetime`` with the
corresponding a :class:`~datetime.timezone` :class:`~datetime.tzinfo`.
.. versionadded:: 3.3
.. function:: mktime_tz(tuple)
Turn a 10-tuple as returned by :func:`parsedate_tz` into a UTC
timestamp (seconds since the Epoch). If the timezone item in the
tuple is ``None``, assume local time.
.. function:: formatdate(timeval=None, localtime=False, usegmt=False)
Returns a date string as per :rfc:`2822`, e.g.::
Fri, 09 Nov 2001 01:08:47 -0000
Optional *timeval* if given is a floating point time value as accepted by
:func:`time.gmtime` and :func:`time.localtime`, otherwise the current time is
used.
Optional *localtime* is a flag that when ``True``, interprets *timeval*, and
returns a date relative to the local timezone instead of UTC, properly taking
daylight savings time into account. The default is ``False`` meaning UTC is
used.
Optional *usegmt* is a flag that when ``True``, outputs a date string with the
timezone as an ascii string ``GMT``, rather than a numeric ``-0000``. This is
needed for some protocols (such as HTTP). This only applies when *localtime* is
``False``. The default is ``False``.
.. function:: format_datetime(dt, usegmt=False)
Like ``formatdate``, but the input is a :mod:`datetime` instance. If it is
a naive datetime, it is assumed to be "UTC with no information about the
source timezone", and the conventional ``-0000`` is used for the timezone.
If it is an aware ``datetime``, then the numeric timezone offset is used.
If it is an aware timezone with offset zero, then *usegmt* may be set to
``True``, in which case the string ``GMT`` is used instead of the numeric
timezone offset. This provides a way to generate standards conformant HTTP
date headers.
.. versionadded:: 3.3
.. function:: decode_rfc2231(s)
Decode the string *s* according to :rfc:`2231`.
.. function:: encode_rfc2231(s, charset=None, language=None)
Encode the string *s* according to :rfc:`2231`. Optional *charset* and
*language*, if given is the character set name and language name to use. If
neither is given, *s* is returned as-is. If *charset* is given but *language*
is not, the string is encoded using the empty string for *language*.
.. function:: collapse_rfc2231_value(value, errors='replace', fallback_charset='us-ascii')
When a header parameter is encoded in :rfc:`2231` format,
:meth:`Message.get_param <email.message.Message.get_param>` may return a
3-tuple containing the character set,
language, and value. :func:`collapse_rfc2231_value` turns this into a unicode
string. Optional *errors* is passed to the *errors* argument of :class:`str`'s
:func:`~str.encode` method; it defaults to ``'replace'``. Optional
*fallback_charset* specifies the character set to use if the one in the
:rfc:`2231` header is not known by Python; it defaults to ``'us-ascii'``.
For convenience, if the *value* passed to :func:`collapse_rfc2231_value` is not
a tuple, it should be a string and it is returned unquoted.
.. function:: decode_params(params)
Decode parameters list according to :rfc:`2231`. *params* is a sequence of
2-tuples containing elements of the form ``(content-type, string-value)``.
.. rubric:: Footnotes
.. [#] Note that the sign of the timezone offset is the opposite of the sign of the
``time.timezone`` variable for the same timezone; the latter variable follows
the POSIX standard while this module follows :rfc:`2822`.

136
third_party/python/Doc/library/ensurepip.rst vendored Normal file
View file

@ -0,0 +1,136 @@
:mod:`ensurepip` --- Bootstrapping the ``pip`` installer
========================================================
.. module:: ensurepip
:synopsis: Bootstrapping the "pip" installer into an existing Python
installation or virtual environment.
.. versionadded:: 3.4
--------------
The :mod:`ensurepip` package provides support for bootstrapping the ``pip``
installer into an existing Python installation or virtual environment. This
bootstrapping approach reflects the fact that ``pip`` is an independent
project with its own release cycle, and the latest available stable version
is bundled with maintenance and feature releases of the CPython reference
interpreter.
In most cases, end users of Python shouldn't need to invoke this module
directly (as ``pip`` should be bootstrapped by default), but it may be
needed if installing ``pip`` was skipped when installing Python (or
when creating a virtual environment) or after explicitly uninstalling
``pip``.
.. note::
This module *does not* access the internet. All of the components
needed to bootstrap ``pip`` are included as internal parts of the
package.
.. seealso::
:ref:`installing-index`
The end user guide for installing Python packages
:pep:`453`: Explicit bootstrapping of pip in Python installations
The original rationale and specification for this module.
Command line interface
----------------------
The command line interface is invoked using the interpreter's ``-m`` switch.
The simplest possible invocation is::
python -m ensurepip
This invocation will install ``pip`` if it is not already installed,
but otherwise does nothing. To ensure the installed version of ``pip``
is at least as recent as the one bundled with ``ensurepip``, pass the
``--upgrade`` option::
python -m ensurepip --upgrade
By default, ``pip`` is installed into the current virtual environment
(if one is active) or into the system site packages (if there is no
active virtual environment). The installation location can be controlled
through two additional command line options:
* ``--root <dir>``: Installs ``pip`` relative to the given root directory
rather than the root of the currently active virtual environment (if any)
or the default root for the current Python installation.
* ``--user``: Installs ``pip`` into the user site packages directory rather
than globally for the current Python installation (this option is not
permitted inside an active virtual environment).
By default, the scripts ``pipX`` and ``pipX.Y`` will be installed (where
X.Y stands for the version of Python used to invoke ``ensurepip``). The
scripts installed can be controlled through two additional command line
options:
* ``--altinstall``: if an alternate installation is requested, the ``pipX``
script will *not* be installed.
* ``--default-pip``: if a "default pip" installation is requested, the
``pip`` script will be installed in addition to the two regular scripts.
Providing both of the script selection options will trigger an exception.
.. versionchanged:: 3.6.3
The exit status is non-zero if the command fails.
Module API
----------
:mod:`ensurepip` exposes two functions for programmatic use:
.. function:: version()
Returns a string specifying the bundled version of pip that will be
installed when bootstrapping an environment.
.. function:: bootstrap(root=None, upgrade=False, user=False, \
altinstall=False, default_pip=False, \
verbosity=0)
Bootstraps ``pip`` into the current or designated environment.
*root* specifies an alternative root directory to install relative to.
If *root* is ``None``, then installation uses the default install location
for the current environment.
*upgrade* indicates whether or not to upgrade an existing installation
of an earlier version of ``pip`` to the bundled version.
*user* indicates whether to use the user scheme rather than installing
globally.
By default, the scripts ``pipX`` and ``pipX.Y`` will be installed (where
X.Y stands for the current version of Python).
If *altinstall* is set, then ``pipX`` will *not* be installed.
If *default_pip* is set, then ``pip`` will be installed in addition to
the two regular scripts.
Setting both *altinstall* and *default_pip* will trigger
:exc:`ValueError`.
*verbosity* controls the level of output to :data:`sys.stdout` from the
bootstrapping operation.
.. note::
The bootstrapping process has side effects on both ``sys.path`` and
``os.environ``. Invoking the command line interface in a subprocess
instead allows these side effects to be avoided.
.. note::
The bootstrapping process may install additional modules required by
``pip``, but other software should not assume those dependencies will
always be present by default (as the dependencies may be removed in a
future version of ``pip``).

1093
third_party/python/Doc/library/enum.rst vendored Normal file
View file

File diff suppressed because it is too large Load diff

639
third_party/python/Doc/library/errno.rst vendored Normal file
View file

@ -0,0 +1,639 @@
:mod:`errno` --- Standard errno system symbols
==============================================
.. module:: errno
:synopsis: Standard errno system symbols.
----------------
This module makes available standard ``errno`` system symbols. The value of each
symbol is the corresponding integer value. The names and descriptions are
borrowed from :file:`linux/include/errno.h`, which should be pretty
all-inclusive.
.. data:: errorcode
Dictionary providing a mapping from the errno value to the string name in the
underlying system. For instance, ``errno.errorcode[errno.EPERM]`` maps to
``'EPERM'``.
To translate a numeric error code to an error message, use :func:`os.strerror`.
Of the following list, symbols that are not used on the current platform are not
defined by the module. The specific list of defined symbols is available as
``errno.errorcode.keys()``. Symbols available can include:
.. data:: EPERM
Operation not permitted
.. data:: ENOENT
No such file or directory
.. data:: ESRCH
No such process
.. data:: EINTR
Interrupted system call.
.. seealso::
This error is mapped to the exception :exc:`InterruptedError`.
.. data:: EIO
I/O error
.. data:: ENXIO
No such device or address
.. data:: E2BIG
Arg list too long
.. data:: ENOEXEC
Exec format error
.. data:: EBADF
Bad file number
.. data:: ECHILD
No child processes
.. data:: EAGAIN
Try again
.. data:: ENOMEM
Out of memory
.. data:: EACCES
Permission denied
.. data:: EFAULT
Bad address
.. data:: ENOTBLK
Block device required
.. data:: EBUSY
Device or resource busy
.. data:: EEXIST
File exists
.. data:: EXDEV
Cross-device link
.. data:: ENODEV
No such device
.. data:: ENOTDIR
Not a directory
.. data:: EISDIR
Is a directory
.. data:: EINVAL
Invalid argument
.. data:: ENFILE
File table overflow
.. data:: EMFILE
Too many open files
.. data:: ENOTTY
Not a typewriter
.. data:: ETXTBSY
Text file busy
.. data:: EFBIG
File too large
.. data:: ENOSPC
No space left on device
.. data:: ESPIPE
Illegal seek
.. data:: EROFS
Read-only file system
.. data:: EMLINK
Too many links
.. data:: EPIPE
Broken pipe
.. data:: EDOM
Math argument out of domain of func
.. data:: ERANGE
Math result not representable
.. data:: EDEADLK
Resource deadlock would occur
.. data:: ENAMETOOLONG
File name too long
.. data:: ENOLCK
No record locks available
.. data:: ENOSYS
Function not implemented
.. data:: ENOTEMPTY
Directory not empty
.. data:: ELOOP
Too many symbolic links encountered
.. data:: EWOULDBLOCK
Operation would block
.. data:: ENOMSG
No message of desired type
.. data:: EIDRM
Identifier removed
.. data:: ECHRNG
Channel number out of range
.. data:: EL2NSYNC
Level 2 not synchronized
.. data:: EL3HLT
Level 3 halted
.. data:: EL3RST
Level 3 reset
.. data:: ELNRNG
Link number out of range
.. data:: EUNATCH
Protocol driver not attached
.. data:: ENOCSI
No CSI structure available
.. data:: EL2HLT
Level 2 halted
.. data:: EBADE
Invalid exchange
.. data:: EBADR
Invalid request descriptor
.. data:: EXFULL
Exchange full
.. data:: ENOANO
No anode
.. data:: EBADRQC
Invalid request code
.. data:: EBADSLT
Invalid slot
.. data:: EDEADLOCK
File locking deadlock error
.. data:: EBFONT
Bad font file format
.. data:: ENOSTR
Device not a stream
.. data:: ENODATA
No data available
.. data:: ETIME
Timer expired
.. data:: ENOSR
Out of streams resources
.. data:: ENONET
Machine is not on the network
.. data:: ENOPKG
Package not installed
.. data:: EREMOTE
Object is remote
.. data:: ENOLINK
Link has been severed
.. data:: EADV
Advertise error
.. data:: ESRMNT
Srmount error
.. data:: ECOMM
Communication error on send
.. data:: EPROTO
Protocol error
.. data:: EMULTIHOP
Multihop attempted
.. data:: EDOTDOT
RFS specific error
.. data:: EBADMSG
Not a data message
.. data:: EOVERFLOW
Value too large for defined data type
.. data:: ENOTUNIQ
Name not unique on network
.. data:: EBADFD
File descriptor in bad state
.. data:: EREMCHG
Remote address changed
.. data:: ELIBACC
Can not access a needed shared library
.. data:: ELIBBAD
Accessing a corrupted shared library
.. data:: ELIBSCN
.lib section in a.out corrupted
.. data:: ELIBMAX
Attempting to link in too many shared libraries
.. data:: ELIBEXEC
Cannot exec a shared library directly
.. data:: EILSEQ
Illegal byte sequence
.. data:: ERESTART
Interrupted system call should be restarted
.. data:: ESTRPIPE
Streams pipe error
.. data:: EUSERS
Too many users
.. data:: ENOTSOCK
Socket operation on non-socket
.. data:: EDESTADDRREQ
Destination address required
.. data:: EMSGSIZE
Message too long
.. data:: EPROTOTYPE
Protocol wrong type for socket
.. data:: ENOPROTOOPT
Protocol not available
.. data:: EPROTONOSUPPORT
Protocol not supported
.. data:: ESOCKTNOSUPPORT
Socket type not supported
.. data:: EOPNOTSUPP
Operation not supported on transport endpoint
.. data:: EPFNOSUPPORT
Protocol family not supported
.. data:: EAFNOSUPPORT
Address family not supported by protocol
.. data:: EADDRINUSE
Address already in use
.. data:: EADDRNOTAVAIL
Cannot assign requested address
.. data:: ENETDOWN
Network is down
.. data:: ENETUNREACH
Network is unreachable
.. data:: ENETRESET
Network dropped connection because of reset
.. data:: ECONNABORTED
Software caused connection abort
.. data:: ECONNRESET
Connection reset by peer
.. data:: ENOBUFS
No buffer space available
.. data:: EISCONN
Transport endpoint is already connected
.. data:: ENOTCONN
Transport endpoint is not connected
.. data:: ESHUTDOWN
Cannot send after transport endpoint shutdown
.. data:: ETOOMANYREFS
Too many references: cannot splice
.. data:: ETIMEDOUT
Connection timed out
.. data:: ECONNREFUSED
Connection refused
.. data:: EHOSTDOWN
Host is down
.. data:: EHOSTUNREACH
No route to host
.. data:: EALREADY
Operation already in progress
.. data:: EINPROGRESS
Operation now in progress
.. data:: ESTALE
Stale NFS file handle
.. data:: EUCLEAN
Structure needs cleaning
.. data:: ENOTNAM
Not a XENIX named type file
.. data:: ENAVAIL
No XENIX semaphores available
.. data:: EISNAM
Is a named type file
.. data:: EREMOTEIO
Remote I/O error
.. data:: EDQUOT
Quota exceeded

737
third_party/python/Doc/library/exceptions.rst vendored Normal file
View file

@ -0,0 +1,737 @@
.. _bltin-exceptions:
Built-in Exceptions
===================
.. index::
statement: try
statement: except
In Python, all exceptions must be instances of a class that derives from
:class:`BaseException`. In a :keyword:`try` statement with an :keyword:`except`
clause that mentions a particular class, that clause also handles any exception
classes derived from that class (but not exception classes from which *it* is
derived). Two exception classes that are not related via subclassing are never
equivalent, even if they have the same name.
.. index:: statement: raise
The built-in exceptions listed below can be generated by the interpreter or
built-in functions. Except where mentioned, they have an "associated value"
indicating the detailed cause of the error. This may be a string or a tuple of
several items of information (e.g., an error code and a string explaining the
code). The associated value is usually passed as arguments to the exception
class's constructor.
User code can raise built-in exceptions. This can be used to test an exception
handler or to report an error condition "just like" the situation in which the
interpreter raises the same exception; but beware that there is nothing to
prevent user code from raising an inappropriate error.
The built-in exception classes can be subclassed to define new exceptions;
programmers are encouraged to derive new exceptions from the :exc:`Exception`
class or one of its subclasses, and not from :exc:`BaseException`. More
information on defining exceptions is available in the Python Tutorial under
:ref:`tut-userexceptions`.
When raising (or re-raising) an exception in an :keyword:`except` or
:keyword:`finally` clause
:attr:`__context__` is automatically set to the last exception caught; if the
new exception is not handled the traceback that is eventually displayed will
include the originating exception(s) and the final exception.
When raising a new exception (rather than using a bare ``raise`` to re-raise
the exception currently being handled), the implicit exception context can be
supplemented with an explicit cause by using :keyword:`from` with
:keyword:`raise`::
raise new_exc from original_exc
The expression following :keyword:`from` must be an exception or ``None``. It
will be set as :attr:`__cause__` on the raised exception. Setting
:attr:`__cause__` also implicitly sets the :attr:`__suppress_context__`
attribute to ``True``, so that using ``raise new_exc from None``
effectively replaces the old exception with the new one for display
purposes (e.g. converting :exc:`KeyError` to :exc:`AttributeError`), while
leaving the old exception available in :attr:`__context__` for introspection
when debugging.
The default traceback display code shows these chained exceptions in
addition to the traceback for the exception itself. An explicitly chained
exception in :attr:`__cause__` is always shown when present. An implicitly
chained exception in :attr:`__context__` is shown only if :attr:`__cause__`
is :const:`None` and :attr:`__suppress_context__` is false.
In either case, the exception itself is always shown after any chained
exceptions so that the final line of the traceback always shows the last
exception that was raised.
Base classes
------------
The following exceptions are used mostly as base classes for other exceptions.
.. exception:: BaseException
The base class for all built-in exceptions. It is not meant to be directly
inherited by user-defined classes (for that, use :exc:`Exception`). If
:func:`str` is called on an instance of this class, the representation of
the argument(s) to the instance are returned, or the empty string when
there were no arguments.
.. attribute:: args
The tuple of arguments given to the exception constructor. Some built-in
exceptions (like :exc:`OSError`) expect a certain number of arguments and
assign a special meaning to the elements of this tuple, while others are
usually called only with a single string giving an error message.
.. method:: with_traceback(tb)
This method sets *tb* as the new traceback for the exception and returns
the exception object. It is usually used in exception handling code like
this::
try:
...
except SomeException:
tb = sys.exc_info()[2]
raise OtherException(...).with_traceback(tb)
.. exception:: Exception
All built-in, non-system-exiting exceptions are derived from this class. All
user-defined exceptions should also be derived from this class.
.. exception:: ArithmeticError
The base class for those built-in exceptions that are raised for various
arithmetic errors: :exc:`OverflowError`, :exc:`ZeroDivisionError`,
:exc:`FloatingPointError`.
.. exception:: BufferError
Raised when a :ref:`buffer <bufferobjects>` related operation cannot be
performed.
.. exception:: LookupError
The base class for the exceptions that are raised when a key or index used on
a mapping or sequence is invalid: :exc:`IndexError`, :exc:`KeyError`. This
can be raised directly by :func:`codecs.lookup`.
Concrete exceptions
-------------------
The following exceptions are the exceptions that are usually raised.
.. exception:: AssertionError
.. index:: statement: assert
Raised when an :keyword:`assert` statement fails.
.. exception:: AttributeError
Raised when an attribute reference (see :ref:`attribute-references`) or
assignment fails. (When an object does not support attribute references or
attribute assignments at all, :exc:`TypeError` is raised.)
.. exception:: EOFError
Raised when the :func:`input` function hits an end-of-file condition (EOF)
without reading any data. (N.B.: the :meth:`io.IOBase.read` and
:meth:`io.IOBase.readline` methods return an empty string when they hit EOF.)
.. exception:: FloatingPointError
Raised when a floating point operation fails. This exception is always defined,
but can only be raised when Python is configured with the
``--with-fpectl`` option, or the :const:`WANT_SIGFPE_HANDLER` symbol is
defined in the :file:`pyconfig.h` file.
.. exception:: GeneratorExit
Raised when a :term:`generator` or :term:`coroutine` is closed;
see :meth:`generator.close` and :meth:`coroutine.close`. It
directly inherits from :exc:`BaseException` instead of :exc:`Exception` since
it is technically not an error.
.. exception:: ImportError
Raised when the :keyword:`import` statement has troubles trying to
load a module. Also raised when the "from list" in ``from ... import``
has a name that cannot be found.
The :attr:`name` and :attr:`path` attributes can be set using keyword-only
arguments to the constructor. When set they represent the name of the module
that was attempted to be imported and the path to any file which triggered
the exception, respectively.
.. versionchanged:: 3.3
Added the :attr:`name` and :attr:`path` attributes.
.. exception:: ModuleNotFoundError
A subclass of :exc:`ImportError` which is raised by :keyword:`import`
when a module could not be located. It is also raised when ``None``
is found in :data:`sys.modules`.
.. versionadded:: 3.6
.. exception:: IndexError
Raised when a sequence subscript is out of range. (Slice indices are
silently truncated to fall in the allowed range; if an index is not an
integer, :exc:`TypeError` is raised.)
.. XXX xref to sequences
.. exception:: KeyError
Raised when a mapping (dictionary) key is not found in the set of existing keys.
.. XXX xref to mapping objects?
.. exception:: KeyboardInterrupt
Raised when the user hits the interrupt key (normally :kbd:`Control-C` or
:kbd:`Delete`). During execution, a check for interrupts is made
regularly. The exception inherits from :exc:`BaseException` so as to not be
accidentally caught by code that catches :exc:`Exception` and thus prevent
the interpreter from exiting.
.. exception:: MemoryError
Raised when an operation runs out of memory but the situation may still be
rescued (by deleting some objects). The associated value is a string indicating
what kind of (internal) operation ran out of memory. Note that because of the
underlying memory management architecture (C's :c:func:`malloc` function), the
interpreter may not always be able to completely recover from this situation; it
nevertheless raises an exception so that a stack traceback can be printed, in
case a run-away program was the cause.
.. exception:: NameError
Raised when a local or global name is not found. This applies only to
unqualified names. The associated value is an error message that includes the
name that could not be found.
.. exception:: NotImplementedError
This exception is derived from :exc:`RuntimeError`. In user defined base
classes, abstract methods should raise this exception when they require
derived classes to override the method, or while the class is being
developed to indicate that the real implementation still needs to be added.
.. note::
It should not be used to indicate that an operator or method is not
meant to be supported at all -- in that case either leave the operator /
method undefined or, if a subclass, set it to :data:`None`.
.. note::
``NotImplementedError`` and ``NotImplemented`` are not interchangeable,
even though they have similar names and purposes. See
:data:`NotImplemented` for details on when to use it.
.. exception:: OSError([arg])
OSError(errno, strerror[, filename[, winerror[, filename2]]])
.. index:: module: errno
This exception is raised when a system function returns a system-related
error, including I/O failures such as "file not found" or "disk full"
(not for illegal argument types or other incidental errors).
The second form of the constructor sets the corresponding attributes,
described below. The attributes default to :const:`None` if not
specified. For backwards compatibility, if three arguments are passed,
the :attr:`~BaseException.args` attribute contains only a 2-tuple
of the first two constructor arguments.
The constructor often actually returns a subclass of :exc:`OSError`, as
described in `OS exceptions`_ below. The particular subclass depends on
the final :attr:`.errno` value. This behaviour only occurs when
constructing :exc:`OSError` directly or via an alias, and is not
inherited when subclassing.
.. attribute:: errno
A numeric error code from the C variable :c:data:`errno`.
.. attribute:: winerror
Under Windows, this gives you the native
Windows error code. The :attr:`.errno` attribute is then an approximate
translation, in POSIX terms, of that native error code.
Under Windows, if the *winerror* constructor argument is an integer,
the :attr:`.errno` attribute is determined from the Windows error code,
and the *errno* argument is ignored. On other platforms, the
*winerror* argument is ignored, and the :attr:`winerror` attribute
does not exist.
.. attribute:: strerror
The corresponding error message, as provided by
the operating system. It is formatted by the C
functions :c:func:`perror` under POSIX, and :c:func:`FormatMessage`
under Windows.
.. attribute:: filename
filename2
For exceptions that involve a file system path (such as :func:`open` or
:func:`os.unlink`), :attr:`filename` is the file name passed to the function.
For functions that involve two file system paths (such as
:func:`os.rename`), :attr:`filename2` corresponds to the second
file name passed to the function.
.. versionchanged:: 3.3
:exc:`EnvironmentError`, :exc:`IOError`, :exc:`WindowsError`,
:exc:`socket.error`, :exc:`select.error` and
:exc:`mmap.error` have been merged into :exc:`OSError`, and the
constructor may return a subclass.
.. versionchanged:: 3.4
The :attr:`filename` attribute is now the original file name passed to
the function, instead of the name encoded to or decoded from the
filesystem encoding. Also, the *filename2* constructor argument and
attribute was added.
.. exception:: OverflowError
Raised when the result of an arithmetic operation is too large to be
represented. This cannot occur for integers (which would rather raise
:exc:`MemoryError` than give up). However, for historical reasons,
OverflowError is sometimes raised for integers that are outside a required
range. Because of the lack of standardization of floating point exception
handling in C, most floating point operations are not checked.
.. exception:: RecursionError
This exception is derived from :exc:`RuntimeError`. It is raised when the
interpreter detects that the maximum recursion depth (see
:func:`sys.getrecursionlimit`) is exceeded.
.. versionadded:: 3.5
Previously, a plain :exc:`RuntimeError` was raised.
.. exception:: ReferenceError
This exception is raised when a weak reference proxy, created by the
:func:`weakref.proxy` function, is used to access an attribute of the referent
after it has been garbage collected. For more information on weak references,
see the :mod:`weakref` module.
.. exception:: RuntimeError
Raised when an error is detected that doesn't fall in any of the other
categories. The associated value is a string indicating what precisely went
wrong.
.. exception:: StopIteration
Raised by built-in function :func:`next` and an :term:`iterator`\'s
:meth:`~iterator.__next__` method to signal that there are no further
items produced by the iterator.
The exception object has a single attribute :attr:`value`, which is
given as an argument when constructing the exception, and defaults
to :const:`None`.
When a :term:`generator` or :term:`coroutine` function
returns, a new :exc:`StopIteration` instance is
raised, and the value returned by the function is used as the
:attr:`value` parameter to the constructor of the exception.
If a generator function defined in the presence of a ``from __future__
import generator_stop`` directive raises :exc:`StopIteration`, it will be
converted into a :exc:`RuntimeError` (retaining the :exc:`StopIteration`
as the new exception's cause).
.. versionchanged:: 3.3
Added ``value`` attribute and the ability for generator functions to
use it to return a value.
.. versionchanged:: 3.5
Introduced the RuntimeError transformation.
.. exception:: StopAsyncIteration
Must be raised by :meth:`__anext__` method of an
:term:`asynchronous iterator` object to stop the iteration.
.. versionadded:: 3.5
.. exception:: SyntaxError
Raised when the parser encounters a syntax error. This may occur in an
:keyword:`import` statement, in a call to the built-in functions :func:`exec`
or :func:`eval`, or when reading the initial script or standard input
(also interactively).
Instances of this class have attributes :attr:`filename`, :attr:`lineno`,
:attr:`offset` and :attr:`text` for easier access to the details. :func:`str`
of the exception instance returns only the message.
.. exception:: IndentationError
Base class for syntax errors related to incorrect indentation. This is a
subclass of :exc:`SyntaxError`.
.. exception:: TabError
Raised when indentation contains an inconsistent use of tabs and spaces.
This is a subclass of :exc:`IndentationError`.
.. exception:: SystemError
Raised when the interpreter finds an internal error, but the situation does not
look so serious to cause it to abandon all hope. The associated value is a
string indicating what went wrong (in low-level terms).
You should report this to the author or maintainer of your Python interpreter.
Be sure to report the version of the Python interpreter (``sys.version``; it is
also printed at the start of an interactive Python session), the exact error
message (the exception's associated value) and if possible the source of the
program that triggered the error.
.. exception:: SystemExit
This exception is raised by the :func:`sys.exit` function. It inherits from
:exc:`BaseException` instead of :exc:`Exception` so that it is not accidentally
caught by code that catches :exc:`Exception`. This allows the exception to
properly propagate up and cause the interpreter to exit. When it is not
handled, the Python interpreter exits; no stack traceback is printed. The
constructor accepts the same optional argument passed to :func:`sys.exit`.
If the value is an integer, it specifies the system exit status (passed to
C's :c:func:`exit` function); if it is ``None``, the exit status is zero; if
it has another type (such as a string), the object's value is printed and
the exit status is one.
A call to :func:`sys.exit` is translated into an exception so that clean-up
handlers (:keyword:`finally` clauses of :keyword:`try` statements) can be
executed, and so that a debugger can execute a script without running the risk
of losing control. The :func:`os._exit` function can be used if it is
absolutely positively necessary to exit immediately (for example, in the child
process after a call to :func:`os.fork`).
.. attribute:: code
The exit status or error message that is passed to the constructor.
(Defaults to ``None``.)
.. exception:: TypeError
Raised when an operation or function is applied to an object of inappropriate
type. The associated value is a string giving details about the type mismatch.
This exception may be raised by user code to indicate that an attempted
operation on an object is not supported, and is not meant to be. If an object
is meant to support a given operation but has not yet provided an
implementation, :exc:`NotImplementedError` is the proper exception to raise.
Passing arguments of the wrong type (e.g. passing a :class:`list` when an
:class:`int` is expected) should result in a :exc:`TypeError`, but passing
arguments with the wrong value (e.g. a number outside expected boundaries)
should result in a :exc:`ValueError`.
.. exception:: UnboundLocalError
Raised when a reference is made to a local variable in a function or method, but
no value has been bound to that variable. This is a subclass of
:exc:`NameError`.
.. exception:: UnicodeError
Raised when a Unicode-related encoding or decoding error occurs. It is a
subclass of :exc:`ValueError`.
:exc:`UnicodeError` has attributes that describe the encoding or decoding
error. For example, ``err.object[err.start:err.end]`` gives the particular
invalid input that the codec failed on.
.. attribute:: encoding
The name of the encoding that raised the error.
.. attribute:: reason
A string describing the specific codec error.
.. attribute:: object
The object the codec was attempting to encode or decode.
.. attribute:: start
The first index of invalid data in :attr:`object`.
.. attribute:: end
The index after the last invalid data in :attr:`object`.
.. exception:: UnicodeEncodeError
Raised when a Unicode-related error occurs during encoding. It is a subclass of
:exc:`UnicodeError`.
.. exception:: UnicodeDecodeError
Raised when a Unicode-related error occurs during decoding. It is a subclass of
:exc:`UnicodeError`.
.. exception:: UnicodeTranslateError
Raised when a Unicode-related error occurs during translating. It is a subclass
of :exc:`UnicodeError`.
.. exception:: ValueError
Raised when an operation or function receives an argument that has the
right type but an inappropriate value, and the situation is not described by a
more precise exception such as :exc:`IndexError`.
.. exception:: ZeroDivisionError
Raised when the second argument of a division or modulo operation is zero. The
associated value is a string indicating the type of the operands and the
operation.
The following exceptions are kept for compatibility with previous versions;
starting from Python 3.3, they are aliases of :exc:`OSError`.
.. exception:: EnvironmentError
.. exception:: IOError
.. exception:: WindowsError
Only available on Windows.
OS exceptions
^^^^^^^^^^^^^
The following exceptions are subclasses of :exc:`OSError`, they get raised
depending on the system error code.
.. exception:: BlockingIOError
Raised when an operation would block on an object (e.g. socket) set
for non-blocking operation.
Corresponds to :c:data:`errno` ``EAGAIN``, ``EALREADY``,
``EWOULDBLOCK`` and ``EINPROGRESS``.
In addition to those of :exc:`OSError`, :exc:`BlockingIOError` can have
one more attribute:
.. attribute:: characters_written
An integer containing the number of characters written to the stream
before it blocked. This attribute is available when using the
buffered I/O classes from the :mod:`io` module.
.. exception:: ChildProcessError
Raised when an operation on a child process failed.
Corresponds to :c:data:`errno` ``ECHILD``.
.. exception:: ConnectionError
A base class for connection-related issues.
Subclasses are :exc:`BrokenPipeError`, :exc:`ConnectionAbortedError`,
:exc:`ConnectionRefusedError` and :exc:`ConnectionResetError`.
.. exception:: BrokenPipeError
A subclass of :exc:`ConnectionError`, raised when trying to write on a
pipe while the other end has been closed, or trying to write on a socket
which has been shutdown for writing.
Corresponds to :c:data:`errno` ``EPIPE`` and ``ESHUTDOWN``.
.. exception:: ConnectionAbortedError
A subclass of :exc:`ConnectionError`, raised when a connection attempt
is aborted by the peer.
Corresponds to :c:data:`errno` ``ECONNABORTED``.
.. exception:: ConnectionRefusedError
A subclass of :exc:`ConnectionError`, raised when a connection attempt
is refused by the peer.
Corresponds to :c:data:`errno` ``ECONNREFUSED``.
.. exception:: ConnectionResetError
A subclass of :exc:`ConnectionError`, raised when a connection is
reset by the peer.
Corresponds to :c:data:`errno` ``ECONNRESET``.
.. exception:: FileExistsError
Raised when trying to create a file or directory which already exists.
Corresponds to :c:data:`errno` ``EEXIST``.
.. exception:: FileNotFoundError
Raised when a file or directory is requested but doesn't exist.
Corresponds to :c:data:`errno` ``ENOENT``.
.. exception:: InterruptedError
Raised when a system call is interrupted by an incoming signal.
Corresponds to :c:data:`errno` :py:data:`~errno.EINTR`.
.. versionchanged:: 3.5
Python now retries system calls when a syscall is interrupted by a
signal, except if the signal handler raises an exception (see :pep:`475`
for the rationale), instead of raising :exc:`InterruptedError`.
.. exception:: IsADirectoryError
Raised when a file operation (such as :func:`os.remove`) is requested
on a directory.
Corresponds to :c:data:`errno` ``EISDIR``.
.. exception:: NotADirectoryError
Raised when a directory operation (such as :func:`os.listdir`) is requested
on something which is not a directory.
Corresponds to :c:data:`errno` ``ENOTDIR``.
.. exception:: PermissionError
Raised when trying to run an operation without the adequate access
rights - for example filesystem permissions.
Corresponds to :c:data:`errno` ``EACCES`` and ``EPERM``.
.. exception:: ProcessLookupError
Raised when a given process doesn't exist.
Corresponds to :c:data:`errno` ``ESRCH``.
.. exception:: TimeoutError
Raised when a system function timed out at the system level.
Corresponds to :c:data:`errno` ``ETIMEDOUT``.
.. versionadded:: 3.3
All the above :exc:`OSError` subclasses were added.
.. seealso::
:pep:`3151` - Reworking the OS and IO exception hierarchy
Warnings
--------
The following exceptions are used as warning categories; see the :mod:`warnings`
module for more information.
.. exception:: Warning
Base class for warning categories.
.. exception:: UserWarning
Base class for warnings generated by user code.
.. exception:: DeprecationWarning
Base class for warnings about deprecated features.
.. exception:: PendingDeprecationWarning
Base class for warnings about features which will be deprecated in the future.
.. exception:: SyntaxWarning
Base class for warnings about dubious syntax.
.. exception:: RuntimeWarning
Base class for warnings about dubious runtime behavior.
.. exception:: FutureWarning
Base class for warnings about constructs that will change semantically in the
future.
.. exception:: ImportWarning
Base class for warnings about probable mistakes in module imports.
.. exception:: UnicodeWarning
Base class for warnings related to Unicode.
.. exception:: BytesWarning
Base class for warnings related to :class:`bytes` and :class:`bytearray`.
.. exception:: ResourceWarning
Base class for warnings related to resource usage.
.. versionadded:: 3.2
Exception hierarchy
-------------------
The class hierarchy for built-in exceptions is:
.. literalinclude:: ../../Lib/test/exception_hierarchy.txt

172
third_party/python/Doc/library/faulthandler.rst vendored Normal file
View file

@ -0,0 +1,172 @@
:mod:`faulthandler` --- Dump the Python traceback
=================================================
.. module:: faulthandler
:synopsis: Dump the Python traceback.
.. versionadded:: 3.3
----------------
This module contains functions to dump Python tracebacks explicitly, on a fault,
after a timeout, or on a user signal. Call :func:`faulthandler.enable` to
install fault handlers for the :const:`SIGSEGV`, :const:`SIGFPE`,
:const:`SIGABRT`, :const:`SIGBUS`, and :const:`SIGILL` signals. You can also
enable them at startup by setting the :envvar:`PYTHONFAULTHANDLER` environment
variable or by using the :option:`-X` ``faulthandler`` command line option.
The fault handler is compatible with system fault handlers like Apport or the
Windows fault handler. The module uses an alternative stack for signal handlers
if the :c:func:`sigaltstack` function is available. This allows it to dump the
traceback even on a stack overflow.
The fault handler is called on catastrophic cases and therefore can only use
signal-safe functions (e.g. it cannot allocate memory on the heap). Because of
this limitation traceback dumping is minimal compared to normal Python
tracebacks:
* Only ASCII is supported. The ``backslashreplace`` error handler is used on
encoding.
* Each string is limited to 500 characters.
* Only the filename, the function name and the line number are
displayed. (no source code)
* It is limited to 100 frames and 100 threads.
* The order is reversed: the most recent call is shown first.
By default, the Python traceback is written to :data:`sys.stderr`. To see
tracebacks, applications must be run in the terminal. A log file can
alternatively be passed to :func:`faulthandler.enable`.
The module is implemented in C, so tracebacks can be dumped on a crash or when
Python is deadlocked.
Dumping the traceback
---------------------
.. function:: dump_traceback(file=sys.stderr, all_threads=True)
Dump the tracebacks of all threads into *file*. If *all_threads* is
``False``, dump only the current thread.
.. versionchanged:: 3.5
Added support for passing file descriptor to this function.
Fault handler state
-------------------
.. function:: enable(file=sys.stderr, all_threads=True)
Enable the fault handler: install handlers for the :const:`SIGSEGV`,
:const:`SIGFPE`, :const:`SIGABRT`, :const:`SIGBUS` and :const:`SIGILL`
signals to dump the Python traceback. If *all_threads* is ``True``,
produce tracebacks for every running thread. Otherwise, dump only the current
thread.
The *file* must be kept open until the fault handler is disabled: see
:ref:`issue with file descriptors <faulthandler-fd>`.
.. versionchanged:: 3.5
Added support for passing file descriptor to this function.
.. versionchanged:: 3.6
On Windows, a handler for Windows exception is also installed.
.. function:: disable()
Disable the fault handler: uninstall the signal handlers installed by
:func:`enable`.
.. function:: is_enabled()
Check if the fault handler is enabled.
Dumping the tracebacks after a timeout
--------------------------------------
.. function:: dump_traceback_later(timeout, repeat=False, file=sys.stderr, exit=False)
Dump the tracebacks of all threads, after a timeout of *timeout* seconds, or
every *timeout* seconds if *repeat* is ``True``. If *exit* is ``True``, call
:c:func:`_exit` with status=1 after dumping the tracebacks. (Note
:c:func:`_exit` exits the process immediately, which means it doesn't do any
cleanup like flushing file buffers.) If the function is called twice, the new
call replaces previous parameters and resets the timeout. The timer has a
sub-second resolution.
The *file* must be kept open until the traceback is dumped or
:func:`cancel_dump_traceback_later` is called: see :ref:`issue with file
descriptors <faulthandler-fd>`.
This function is implemented using a watchdog thread and therefore is not
available if Python is compiled with threads disabled.
.. versionchanged:: 3.5
Added support for passing file descriptor to this function.
.. function:: cancel_dump_traceback_later()
Cancel the last call to :func:`dump_traceback_later`.
Dumping the traceback on a user signal
--------------------------------------
.. function:: register(signum, file=sys.stderr, all_threads=True, chain=False)
Register a user signal: install a handler for the *signum* signal to dump
the traceback of all threads, or of the current thread if *all_threads* is
``False``, into *file*. Call the previous handler if chain is ``True``.
The *file* must be kept open until the signal is unregistered by
:func:`unregister`: see :ref:`issue with file descriptors <faulthandler-fd>`.
Not available on Windows.
.. versionchanged:: 3.5
Added support for passing file descriptor to this function.
.. function:: unregister(signum)
Unregister a user signal: uninstall the handler of the *signum* signal
installed by :func:`register`. Return ``True`` if the signal was registered,
``False`` otherwise.
Not available on Windows.
.. _faulthandler-fd:
Issue with file descriptors
---------------------------
:func:`enable`, :func:`dump_traceback_later` and :func:`register` keep the
file descriptor of their *file* argument. If the file is closed and its file
descriptor is reused by a new file, or if :func:`os.dup2` is used to replace
the file descriptor, the traceback will be written into a different file. Call
these functions again each time that the file is replaced.
Example
-------
Example of a segmentation fault on Linux with and without enabling the fault
handler:
.. code-block:: shell-session
$ python3 -c "import ctypes; ctypes.string_at(0)"
Segmentation fault
$ python3 -q -X faulthandler
>>> import ctypes
>>> ctypes.string_at(0)
Fatal Python error: Segmentation fault
Current thread 0x00007fb899f39700 (most recent call first):
File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at
File "<stdin>", line 1 in <module>
Segmentation fault

170
third_party/python/Doc/library/fcntl.rst vendored Normal file
View file

@ -0,0 +1,170 @@
:mod:`fcntl` --- The ``fcntl`` and ``ioctl`` system calls
=========================================================
.. module:: fcntl
:platform: Unix
:synopsis: The fcntl() and ioctl() system calls.
.. sectionauthor:: Jaap Vermeulen
.. index::
pair: UNIX; file control
pair: UNIX; I/O control
----------------
This module performs file control and I/O control on file descriptors. It is an
interface to the :c:func:`fcntl` and :c:func:`ioctl` Unix routines. For a
complete description of these calls, see :manpage:`fcntl(2)` and
:manpage:`ioctl(2)` Unix manual pages.
All functions in this module take a file descriptor *fd* as their first
argument. This can be an integer file descriptor, such as returned by
``sys.stdin.fileno()``, or an :class:`io.IOBase` object, such as ``sys.stdin``
itself, which provides a :meth:`~io.IOBase.fileno` that returns a genuine file
descriptor.
.. versionchanged:: 3.3
Operations in this module used to raise an :exc:`IOError` where they now
raise an :exc:`OSError`.
The module defines the following functions:
.. function:: fcntl(fd, cmd, arg=0)
Perform the operation *cmd* on file descriptor *fd* (file objects providing
a :meth:`~io.IOBase.fileno` method are accepted as well). The values used
for *cmd* are operating system dependent, and are available as constants
in the :mod:`fcntl` module, using the same names as used in the relevant C
header files. The argument *arg* can either be an integer value, or a
:class:`bytes` object. With an integer value, the return value of this
function is the integer return value of the C :c:func:`fcntl` call. When
the argument is bytes it represents a binary structure, e.g. created by
:func:`struct.pack`. The binary data is copied to a buffer whose address is
passed to the C :c:func:`fcntl` call. The return value after a successful
call is the contents of the buffer, converted to a :class:`bytes` object.
The length of the returned object will be the same as the length of the
*arg* argument. This is limited to 1024 bytes. If the information returned
in the buffer by the operating system is larger than 1024 bytes, this is
most likely to result in a segmentation violation or a more subtle data
corruption.
If the :c:func:`fcntl` fails, an :exc:`OSError` is raised.
.. function:: ioctl(fd, request, arg=0, mutate_flag=True)
This function is identical to the :func:`~fcntl.fcntl` function, except
that the argument handling is even more complicated.
The *request* parameter is limited to values that can fit in 32-bits.
Additional constants of interest for use as the *request* argument can be
found in the :mod:`termios` module, under the same names as used in
the relevant C header files.
The parameter *arg* can be one of an integer, an object supporting the
read-only buffer interface (like :class:`bytes`) or an object supporting
the read-write buffer interface (like :class:`bytearray`).
In all but the last case, behaviour is as for the :func:`~fcntl.fcntl`
function.
If a mutable buffer is passed, then the behaviour is determined by the value of
the *mutate_flag* parameter.
If it is false, the buffer's mutability is ignored and behaviour is as for a
read-only buffer, except that the 1024 byte limit mentioned above is avoided --
so long as the buffer you pass is at least as long as what the operating system
wants to put there, things should work.
If *mutate_flag* is true (the default), then the buffer is (in effect) passed
to the underlying :func:`ioctl` system call, the latter's return code is
passed back to the calling Python, and the buffer's new contents reflect the
action of the :func:`ioctl`. This is a slight simplification, because if the
supplied buffer is less than 1024 bytes long it is first copied into a static
buffer 1024 bytes long which is then passed to :func:`ioctl` and copied back
into the supplied buffer.
If the :c:func:`ioctl` fails, an :exc:`OSError` exception is raised.
An example::
>>> import array, fcntl, struct, termios, os
>>> os.getpgrp()
13341
>>> struct.unpack('h', fcntl.ioctl(0, termios.TIOCGPGRP, " "))[0]
13341
>>> buf = array.array('h', [0])
>>> fcntl.ioctl(0, termios.TIOCGPGRP, buf, 1)
0
>>> buf
array('h', [13341])
.. function:: flock(fd, operation)
Perform the lock operation *operation* on file descriptor *fd* (file objects providing
a :meth:`~io.IOBase.fileno` method are accepted as well). See the Unix manual
:manpage:`flock(2)` for details. (On some systems, this function is emulated
using :c:func:`fcntl`.)
If the :c:func:`flock` fails, an :exc:`OSError` exception is raised.
.. function:: lockf(fd, cmd, len=0, start=0, whence=0)
This is essentially a wrapper around the :func:`~fcntl.fcntl` locking calls.
*fd* is the file descriptor of the file to lock or unlock, and *cmd*
is one of the following values:
* :const:`LOCK_UN` -- unlock
* :const:`LOCK_SH` -- acquire a shared lock
* :const:`LOCK_EX` -- acquire an exclusive lock
When *cmd* is :const:`LOCK_SH` or :const:`LOCK_EX`, it can also be
bitwise ORed with :const:`LOCK_NB` to avoid blocking on lock acquisition.
If :const:`LOCK_NB` is used and the lock cannot be acquired, an
:exc:`OSError` will be raised and the exception will have an *errno*
attribute set to :const:`EACCES` or :const:`EAGAIN` (depending on the
operating system; for portability, check for both values). On at least some
systems, :const:`LOCK_EX` can only be used if the file descriptor refers to a
file opened for writing.
*len* is the number of bytes to lock, *start* is the byte offset at
which the lock starts, relative to *whence*, and *whence* is as with
:func:`io.IOBase.seek`, specifically:
* :const:`0` -- relative to the start of the file (:data:`os.SEEK_SET`)
* :const:`1` -- relative to the current buffer position (:data:`os.SEEK_CUR`)
* :const:`2` -- relative to the end of the file (:data:`os.SEEK_END`)
The default for *start* is 0, which means to start at the beginning of the file.
The default for *len* is 0 which means to lock to the end of the file. The
default for *whence* is also 0.
Examples (all on a SVR4 compliant system)::
import struct, fcntl, os
f = open(...)
rv = fcntl.fcntl(f, fcntl.F_SETFL, os.O_NDELAY)
lockdata = struct.pack('hhllhh', fcntl.F_WRLCK, 0, 0, 0, 0, 0)
rv = fcntl.fcntl(f, fcntl.F_SETLKW, lockdata)
Note that in the first example the return value variable *rv* will hold an
integer value; in the second example it will hold a :class:`bytes` object. The
structure lay-out for the *lockdata* variable is system dependent --- therefore
using the :func:`flock` call may be better.
.. seealso::
Module :mod:`os`
If the locking flags :data:`~os.O_SHLOCK` and :data:`~os.O_EXLOCK` are
present in the :mod:`os` module (on BSD only), the :func:`os.open`
function provides an alternative to the :func:`lockf` and :func:`flock`
functions.

198
third_party/python/Doc/library/filecmp.rst vendored Normal file
View file

@ -0,0 +1,198 @@
:mod:`filecmp` --- File and Directory Comparisons
=================================================
.. module:: filecmp
:synopsis: Compare files efficiently.
.. sectionauthor:: Moshe Zadka <moshez@zadka.site.co.il>
**Source code:** :source:`Lib/filecmp.py`
--------------
The :mod:`filecmp` module defines functions to compare files and directories,
with various optional time/correctness trade-offs. For comparing files,
see also the :mod:`difflib` module.
The :mod:`filecmp` module defines the following functions:
.. function:: cmp(f1, f2, shallow=True)
Compare the files named *f1* and *f2*, returning ``True`` if they seem equal,
``False`` otherwise.
If *shallow* is true, files with identical :func:`os.stat` signatures are
taken to be equal. Otherwise, the contents of the files are compared.
Note that no external programs are called from this function, giving it
portability and efficiency.
This function uses a cache for past comparisons and the results,
with cache entries invalidated if the :func:`os.stat` information for the
file changes. The entire cache may be cleared using :func:`clear_cache`.
.. function:: cmpfiles(dir1, dir2, common, shallow=True)
Compare the files in the two directories *dir1* and *dir2* whose names are
given by *common*.
Returns three lists of file names: *match*, *mismatch*,
*errors*. *match* contains the list of files that match, *mismatch* contains
the names of those that don't, and *errors* lists the names of files which
could not be compared. Files are listed in *errors* if they don't exist in
one of the directories, the user lacks permission to read them or if the
comparison could not be done for some other reason.
The *shallow* parameter has the same meaning and default value as for
:func:`filecmp.cmp`.
For example, ``cmpfiles('a', 'b', ['c', 'd/e'])`` will compare ``a/c`` with
``b/c`` and ``a/d/e`` with ``b/d/e``. ``'c'`` and ``'d/e'`` will each be in
one of the three returned lists.
.. function:: clear_cache()
Clear the filecmp cache. This may be useful if a file is compared so quickly
after it is modified that it is within the mtime resolution of
the underlying filesystem.
.. versionadded:: 3.4
.. _dircmp-objects:
The :class:`dircmp` class
-------------------------
.. class:: dircmp(a, b, ignore=None, hide=None)
Construct a new directory comparison object, to compare the directories *a*
and *b*. *ignore* is a list of names to ignore, and defaults to
:attr:`filecmp.DEFAULT_IGNORES`. *hide* is a list of names to hide, and
defaults to ``[os.curdir, os.pardir]``.
The :class:`dircmp` class compares files by doing *shallow* comparisons
as described for :func:`filecmp.cmp`.
The :class:`dircmp` class provides the following methods:
.. method:: report()
Print (to :data:`sys.stdout`) a comparison between *a* and *b*.
.. method:: report_partial_closure()
Print a comparison between *a* and *b* and common immediate
subdirectories.
.. method:: report_full_closure()
Print a comparison between *a* and *b* and common subdirectories
(recursively).
The :class:`dircmp` class offers a number of interesting attributes that may be
used to get various bits of information about the directory trees being
compared.
Note that via :meth:`__getattr__` hooks, all attributes are computed lazily,
so there is no speed penalty if only those attributes which are lightweight
to compute are used.
.. attribute:: left
The directory *a*.
.. attribute:: right
The directory *b*.
.. attribute:: left_list
Files and subdirectories in *a*, filtered by *hide* and *ignore*.
.. attribute:: right_list
Files and subdirectories in *b*, filtered by *hide* and *ignore*.
.. attribute:: common
Files and subdirectories in both *a* and *b*.
.. attribute:: left_only
Files and subdirectories only in *a*.
.. attribute:: right_only
Files and subdirectories only in *b*.
.. attribute:: common_dirs
Subdirectories in both *a* and *b*.
.. attribute:: common_files
Files in both *a* and *b*.
.. attribute:: common_funny
Names in both *a* and *b*, such that the type differs between the
directories, or names for which :func:`os.stat` reports an error.
.. attribute:: same_files
Files which are identical in both *a* and *b*, using the class's
file comparison operator.
.. attribute:: diff_files
Files which are in both *a* and *b*, whose contents differ according
to the class's file comparison operator.
.. attribute:: funny_files
Files which are in both *a* and *b*, but could not be compared.
.. attribute:: subdirs
A dictionary mapping names in :attr:`common_dirs` to :class:`dircmp`
objects.
.. attribute:: DEFAULT_IGNORES
.. versionadded:: 3.4
List of directories ignored by :class:`dircmp` by default.
Here is a simplified example of using the ``subdirs`` attribute to search
recursively through two directories to show common different files::
>>> from filecmp import dircmp
>>> def print_diff_files(dcmp):
... for name in dcmp.diff_files:
... print("diff_file %s found in %s and %s" % (name, dcmp.left,
... dcmp.right))
... for sub_dcmp in dcmp.subdirs.values():
... print_diff_files(sub_dcmp)
...
>>> dcmp = dircmp('dir1', 'dir2') # doctest: +SKIP
>>> print_diff_files(dcmp) # doctest: +SKIP

17
third_party/python/Doc/library/fileformats.rst vendored Normal file
View file

@ -0,0 +1,17 @@
.. _fileformats:
************
File Formats
************
The modules described in this chapter parse various miscellaneous file formats
that aren't markup languages and are not related to e-mail.
.. toctree::
csv.rst
configparser.rst
netrc.rst
xdrlib.rst
plistlib.rst

207
third_party/python/Doc/library/fileinput.rst vendored Normal file
View file

@ -0,0 +1,207 @@
:mod:`fileinput` --- Iterate over lines from multiple input streams
===================================================================
.. module:: fileinput
:synopsis: Loop over standard input or a list of files.
.. moduleauthor:: Guido van Rossum <guido@python.org>
.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
**Source code:** :source:`Lib/fileinput.py`
--------------
This module implements a helper class and functions to quickly write a
loop over standard input or a list of files. If you just want to read or
write one file see :func:`open`.
The typical use is::
import fileinput
for line in fileinput.input():
process(line)
This iterates over the lines of all files listed in ``sys.argv[1:]``, defaulting
to ``sys.stdin`` if the list is empty. If a filename is ``'-'``, it is also
replaced by ``sys.stdin``. To specify an alternative list of filenames, pass it
as the first argument to :func:`.input`. A single file name is also allowed.
All files are opened in text mode by default, but you can override this by
specifying the *mode* parameter in the call to :func:`.input` or
:class:`FileInput`. If an I/O error occurs during opening or reading a file,
:exc:`OSError` is raised.
.. versionchanged:: 3.3
:exc:`IOError` used to be raised; it is now an alias of :exc:`OSError`.
If ``sys.stdin`` is used more than once, the second and further use will return
no lines, except perhaps for interactive use, or if it has been explicitly reset
(e.g. using ``sys.stdin.seek(0)``).
Empty files are opened and immediately closed; the only time their presence in
the list of filenames is noticeable at all is when the last file opened is
empty.
Lines are returned with any newlines intact, which means that the last line in
a file may not have one.
You can control how files are opened by providing an opening hook via the
*openhook* parameter to :func:`fileinput.input` or :class:`FileInput()`. The
hook must be a function that takes two arguments, *filename* and *mode*, and
returns an accordingly opened file-like object. Two useful hooks are already
provided by this module.
The following function is the primary interface of this module:
.. function:: input(files=None, inplace=False, backup='', bufsize=0, mode='r', openhook=None)
Create an instance of the :class:`FileInput` class. The instance will be used
as global state for the functions of this module, and is also returned to use
during iteration. The parameters to this function will be passed along to the
constructor of the :class:`FileInput` class.
The :class:`FileInput` instance can be used as a context manager in the
:keyword:`with` statement. In this example, *input* is closed after the
:keyword:`with` statement is exited, even if an exception occurs::
with fileinput.input(files=('spam.txt', 'eggs.txt')) as f:
for line in f:
process(line)
.. versionchanged:: 3.2
Can be used as a context manager.
.. deprecated-removed:: 3.6 3.8
The *bufsize* parameter.
The following functions use the global state created by :func:`fileinput.input`;
if there is no active state, :exc:`RuntimeError` is raised.
.. function:: filename()
Return the name of the file currently being read. Before the first line has
been read, returns ``None``.
.. function:: fileno()
Return the integer "file descriptor" for the current file. When no file is
opened (before the first line and between files), returns ``-1``.
.. function:: lineno()
Return the cumulative line number of the line that has just been read. Before
the first line has been read, returns ``0``. After the last line of the last
file has been read, returns the line number of that line.
.. function:: filelineno()
Return the line number in the current file. Before the first line has been
read, returns ``0``. After the last line of the last file has been read,
returns the line number of that line within the file.
.. function:: isfirstline()
Returns true if the line just read is the first line of its file, otherwise
returns false.
.. function:: isstdin()
Returns true if the last line was read from ``sys.stdin``, otherwise returns
false.
.. function:: nextfile()
Close the current file so that the next iteration will read the first line from
the next file (if any); lines not read from the file will not count towards the
cumulative line count. The filename is not changed until after the first line
of the next file has been read. Before the first line has been read, this
function has no effect; it cannot be used to skip the first file. After the
last line of the last file has been read, this function has no effect.
.. function:: close()
Close the sequence.
The class which implements the sequence behavior provided by the module is
available for subclassing as well:
.. class:: FileInput(files=None, inplace=False, backup='', bufsize=0, mode='r', openhook=None)
Class :class:`FileInput` is the implementation; its methods :meth:`filename`,
:meth:`fileno`, :meth:`lineno`, :meth:`filelineno`, :meth:`isfirstline`,
:meth:`isstdin`, :meth:`nextfile` and :meth:`close` correspond to the
functions of the same name in the module. In addition it has a
:meth:`~io.TextIOBase.readline` method which returns the next input line,
and a :meth:`__getitem__` method which implements the sequence behavior.
The sequence must be accessed in strictly sequential order; random access
and :meth:`~io.TextIOBase.readline` cannot be mixed.
With *mode* you can specify which file mode will be passed to :func:`open`. It
must be one of ``'r'``, ``'rU'``, ``'U'`` and ``'rb'``.
The *openhook*, when given, must be a function that takes two arguments,
*filename* and *mode*, and returns an accordingly opened file-like object. You
cannot use *inplace* and *openhook* together.
A :class:`FileInput` instance can be used as a context manager in the
:keyword:`with` statement. In this example, *input* is closed after the
:keyword:`with` statement is exited, even if an exception occurs::
with FileInput(files=('spam.txt', 'eggs.txt')) as input:
process(input)
.. versionchanged:: 3.2
Can be used as a context manager.
.. deprecated:: 3.4
The ``'rU'`` and ``'U'`` modes.
.. deprecated-removed:: 3.6 3.8
The *bufsize* parameter.
**Optional in-place filtering:** if the keyword argument ``inplace=True`` is
passed to :func:`fileinput.input` or to the :class:`FileInput` constructor, the
file is moved to a backup file and standard output is directed to the input file
(if a file of the same name as the backup file already exists, it will be
replaced silently). This makes it possible to write a filter that rewrites its
input file in place. If the *backup* parameter is given (typically as
``backup='.<some extension>'``), it specifies the extension for the backup file,
and the backup file remains around; by default, the extension is ``'.bak'`` and
it is deleted when the output file is closed. In-place filtering is disabled
when standard input is read.
The two following opening hooks are provided by this module:
.. function:: hook_compressed(filename, mode)
Transparently opens files compressed with gzip and bzip2 (recognized by the
extensions ``'.gz'`` and ``'.bz2'``) using the :mod:`gzip` and :mod:`bz2`
modules. If the filename extension is not ``'.gz'`` or ``'.bz2'``, the file is
opened normally (ie, using :func:`open` without any decompression).
Usage example: ``fi = fileinput.FileInput(openhook=fileinput.hook_compressed)``
.. function:: hook_encoded(encoding, errors=None)
Returns a hook which opens each file with :func:`open`, using the given
*encoding* and *errors* to read the file.
Usage example: ``fi =
fileinput.FileInput(openhook=fileinput.hook_encoded("utf-8",
"surrogateescape"))``
.. versionchanged:: 3.6
Added the optional *errors* parameter.

Some files were not shown because too many files have changed in this diff Show more