Undiamond Python headers

This change gets the Python codebase into a state where it conforms to
the conventions of this codebase. It's now possible to include headers
from Python, without worrying about ordering. Python has traditionally
solved that problem by "diamonding" everything in Python.h, but that's
problematic since it means any change to any Python header invalidates
all the build artifacts. Lastly it makes tooling not work. Since it is
hard to explain to Emacs when I press C-c C-h to add an import line it
shouldn't add the header that actually defines the symbol, and instead
do follow the nonstandard Python convention.

Progress has been made on letting Python load source code from the zip
executable structure via the standard C library APIs. System calss now
recognizes zip!FILENAME alternative URIs as equivalent to zip:FILENAME
since Python uses colon as its delimiter.

Some progress has been made on embedding the notice license terms into
the Python object code. This is easier said than done since Python has
an extremely complicated ownership story.

- Some termios APIs have been added
- Implement rewinddir() dirstream API
- GetCpuCount() API added to Cosmopolitan Libc
- More bugs in Cosmopolitan Libc have been fixed
- zipobj.com now has flags for mangling the path
- Fixed bug a priori with sendfile() on certain BSDs
- Polyfill F_DUPFD and F_DUPFD_CLOEXEC across platforms
- FIOCLEX / FIONCLEX now polyfilled for fast O_CLOEXEC changes
- APE now supports a hybrid solution to no-self-modify for builds
- Many BSD-only magnums added, e.g. O_SEARCH, O_SHLOCK, SF_NODISKIO
This commit is contained in:
Justine Tunney 2021-08-12 00:42:14 -07:00
parent 20bb8db9f8
commit b420ed8248
762 changed files with 18410 additions and 53772 deletions

View file

@ -1,108 +1,30 @@
#ifndef Py_OBJIMPL_H
#define Py_OBJIMPL_H
#include "libc/assert.h"
#include "third_party/python/Include/object.h"
#include "third_party/python/Include/pyerrors.h"
#include "third_party/python/Include/pymacro.h"
#include "third_party/python/Include/pymem.h"
#include "third_party/python/Include/tupleobject.h"
COSMOPOLITAN_C_START_
/* clang-format off */
/* BEWARE:
Each interface exports both functions and macros. Extension modules should
use the functions, to ensure binary compatibility across Python versions.
Because the Python implementation is free to change internal details, and
the macros may (or may not) expose details for speed, if you do use the
macros you must recompile your extensions with each Python release.
Never mix calls to PyObject_ memory functions with calls to the platform
malloc/realloc/ calloc/free, or with calls to PyMem_.
*/
/*
Functions and macros for modules that implement new object types.
- PyObject_New(type, typeobj) allocates memory for a new object of the given
type, and initializes part of it. 'type' must be the C structure type used
to represent the object, and 'typeobj' the address of the corresponding
type object. Reference count and type pointer are filled in; the rest of
the bytes of the object are *undefined*! The resulting expression type is
'type *'. The size of the object is determined by the tp_basicsize field
of the type object.
- PyObject_NewVar(type, typeobj, n) is similar but allocates a variable-size
object with room for n items. In addition to the refcount and type pointer
fields, this also fills in the ob_size field.
- PyObject_Del(op) releases the memory allocated for an object. It does not
run a destructor -- it only frees the memory. PyObject_Free is identical.
- PyObject_Init(op, typeobj) and PyObject_InitVar(op, typeobj, n) don't
allocate memory. Instead of a 'type' parameter, they take a pointer to a
new object (allocated by an arbitrary allocator), and initialize its object
header fields.
Note that objects created with PyObject_{New, NewVar} are allocated using the
specialized Python allocator (implemented in obmalloc.c), if WITH_PYMALLOC is
enabled. In addition, a special debugging allocator is used if PYMALLOC_DEBUG
is also #defined.
In case a specific form of memory management is needed (for example, if you
must use the platform malloc heap(s), or shared memory, or C++ local storage or
operator new), you must first allocate the object with your custom allocator,
then pass its pointer to PyObject_{Init, InitVar} for filling in its Python-
specific fields: reference count, type pointer, possibly others. You should
be aware that Python has no control over these objects because they don't
cooperate with the Python memory manager. Such objects may not be eligible
for automatic garbage collection and you have to make sure that they are
released accordingly whenever their destructor gets called (cf. the specific
form of memory management you're using).
Unless you have specific memory management requirements, use
PyObject_{New, NewVar, Del}.
*/
/*
* Raw object memory interface
* ===========================
*/
/* Functions to call the same malloc/realloc/free as used by Python's
object allocator. If WITH_PYMALLOC is enabled, these may differ from
the platform malloc/realloc/free. The Python object allocator is
designed for fast, cache-conscious allocation of many "small" objects,
and with low hidden memory overhead.
PyObject_Malloc(0) returns a unique non-NULL pointer if possible.
PyObject_Realloc(NULL, n) acts like PyObject_Malloc(n).
PyObject_Realloc(p != NULL, 0) does not return NULL, or free the memory
at p.
Returned pointers must be checked for NULL explicitly; no action is
performed on failure other than to return NULL (no warning it printed, no
exception is set, etc).
For allocating objects, use PyObject_{New, NewVar} instead whenever
possible. The PyObject_{Malloc, Realloc, Free} family is exposed
so that you can exploit Python's small-block allocator for non-object
uses. If you must use these routines to allocate object memory, make sure
the object gets initialized via PyObject_{Init, InitVar} after obtaining
the raw memory.
*/
PyAPI_FUNC(void *) PyObject_Malloc(size_t size);
void * PyObject_Malloc(size_t size);
#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03050000
PyAPI_FUNC(void *) PyObject_Calloc(size_t nelem, size_t elsize);
void * PyObject_Calloc(size_t nelem, size_t elsize);
#endif
PyAPI_FUNC(void *) PyObject_Realloc(void *ptr, size_t new_size);
PyAPI_FUNC(void) PyObject_Free(void *ptr);
void * PyObject_Realloc(void *ptr, size_t new_size);
void PyObject_Free(void *ptr);
#ifndef Py_LIMITED_API
/* This function returns the number of allocated memory blocks, regardless of size */
PyAPI_FUNC(Py_ssize_t) _Py_GetAllocatedBlocks(void);
Py_ssize_t _Py_GetAllocatedBlocks(void);
#endif /* !Py_LIMITED_API */
/* Macros */
#ifdef WITH_PYMALLOC
#ifndef Py_LIMITED_API
PyAPI_FUNC(void) _PyObject_DebugMallocStats(FILE *out);
void _PyObject_DebugMallocStats(FILE *out);
#endif /* #ifndef Py_LIMITED_API */
#endif
@ -120,11 +42,11 @@ PyAPI_FUNC(void) _PyObject_DebugMallocStats(FILE *out);
*/
/* Functions */
PyAPI_FUNC(PyObject *) PyObject_Init(PyObject *, PyTypeObject *);
PyAPI_FUNC(PyVarObject *) PyObject_InitVar(PyVarObject *,
PyObject * PyObject_Init(PyObject *, PyTypeObject *);
PyVarObject * PyObject_InitVar(PyVarObject *,
PyTypeObject *, Py_ssize_t);
PyAPI_FUNC(PyObject *) _PyObject_New(PyTypeObject *);
PyAPI_FUNC(PyVarObject *) _PyObject_NewVar(PyTypeObject *, Py_ssize_t);
PyObject * _PyObject_New(PyTypeObject *);
PyVarObject * _PyObject_NewVar(PyTypeObject *, Py_ssize_t);
#define PyObject_New(type, typeobj) \
( (type *) _PyObject_New(typeobj) )
@ -209,10 +131,10 @@ typedef struct {
} PyObjectArenaAllocator;
/* Get the arena allocator. */
PyAPI_FUNC(void) PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator);
void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator);
/* Set the arena allocator. */
PyAPI_FUNC(void) PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator);
void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator);
#endif
@ -222,11 +144,11 @@ PyAPI_FUNC(void) PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator);
*/
/* C equivalent of gc.collect() which ignores the state of gc.enabled. */
PyAPI_FUNC(Py_ssize_t) PyGC_Collect(void);
Py_ssize_t PyGC_Collect(void);
#ifndef Py_LIMITED_API
PyAPI_FUNC(Py_ssize_t) _PyGC_CollectNoFail(void);
PyAPI_FUNC(Py_ssize_t) _PyGC_CollectIfEnabled(void);
Py_ssize_t _PyGC_CollectNoFail(void);
Py_ssize_t _PyGC_CollectIfEnabled(void);
#endif
/* Test if a type has a GC head */
@ -236,7 +158,7 @@ PyAPI_FUNC(Py_ssize_t) _PyGC_CollectIfEnabled(void);
#define PyObject_IS_GC(o) (PyType_IS_GC(Py_TYPE(o)) && \
(Py_TYPE(o)->tp_is_gc == NULL || Py_TYPE(o)->tp_is_gc(o)))
PyAPI_FUNC(PyVarObject *) _PyObject_GC_Resize(PyVarObject *, Py_ssize_t);
PyVarObject * _PyObject_GC_Resize(PyVarObject *, Py_ssize_t);
#define PyObject_GC_Resize(type, op, n) \
( (type *) _PyObject_GC_Resize((PyVarObject *)(op), (n)) )
@ -321,14 +243,14 @@ extern PyGC_Head *_PyGC_generation0;
#endif /* Py_LIMITED_API */
#ifndef Py_LIMITED_API
PyAPI_FUNC(PyObject *) _PyObject_GC_Malloc(size_t size);
PyAPI_FUNC(PyObject *) _PyObject_GC_Calloc(size_t size);
PyObject * _PyObject_GC_Malloc(size_t size);
PyObject * _PyObject_GC_Calloc(size_t size);
#endif /* !Py_LIMITED_API */
PyAPI_FUNC(PyObject *) _PyObject_GC_New(PyTypeObject *);
PyAPI_FUNC(PyVarObject *) _PyObject_GC_NewVar(PyTypeObject *, Py_ssize_t);
PyAPI_FUNC(void) PyObject_GC_Track(void *);
PyAPI_FUNC(void) PyObject_GC_UnTrack(void *);
PyAPI_FUNC(void) PyObject_GC_Del(void *);
PyObject * _PyObject_GC_New(PyTypeObject *);
PyVarObject * _PyObject_GC_NewVar(PyTypeObject *, Py_ssize_t);
void PyObject_GC_Track(void *);
void PyObject_GC_UnTrack(void *);
void PyObject_GC_Del(void *);
#define PyObject_GC_New(type, typeobj) \
( (type *) _PyObject_GC_New(typeobj) )