gh-145921: Add "_DuringGC" functions for tp_traverse (GH-145925)

There are newly documented restrictions on tp_traverse:

    The traversal function must not have any side effects.
    It must not modify the reference counts of any Python
    objects nor create or destroy any Python objects.

* Add several functions that are guaranteed side-effect-free,
  with a _DuringGC suffix.
* Use these in ctypes
* Consolidate tp_traverse docs in gcsupport.rst, moving unique
  content from typeobj.rst there

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>

Author: 3 people

Doc/c-api/gcsupport.rst

@@ -220,54 +220,237 @@ The :c:member:`~PyTypeObject.tp_traverse` handler accepts a function parameter o
    detection; it's not expected that users will need to write their own
    visitor functions.
 
-The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type:
+The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or ``NULL``
+if the object is immutable.
+
+
+.. c:type:: int (*inquiry)(PyObject *self)
+
+   Drop references that may have created reference cycles.  Immutable objects
+   do not have to define this method since they can never directly create
+   reference cycles.  Note that the object must still be valid after calling
+   this method (don't just call :c:func:`Py_DECREF` on a reference).  The
+   collector will call this method if it detects that this object is involved
+   in a reference cycle.
+
+
+.. _gc-traversal:
 
+Traversal
+---------
+
+The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type:
 
 .. c:type:: int (*traverseproc)(PyObject *self, visitproc visit, void *arg)
 
-   Traversal function for a container object.  Implementations must call the
+   Traversal function for a garbage-collected object, used by the garbage
+   collector to detect reference cycles.
+   Implementations must call the
    *visit* function for each object directly contained by *self*, with the
    parameters to *visit* being the contained object and the *arg* value passed
    to the handler.  The *visit* function must not be called with a ``NULL``
-   object argument.  If *visit* returns a non-zero value that value should be
+   object argument.  If *visit* returns a non-zero value, that value should be
    returned immediately.
 
-   The traversal function must not have any side effects.  Implementations
-   may not modify the reference counts of any Python objects nor create or
-   destroy any Python objects.
+   A typical :c:member:`!tp_traverse` function calls the :c:func:`Py_VISIT`
+   convenience macro on each of the instance's members that are Python
+   objects that the instance owns.
+   For example, this is a (slightly outdated) traversal function for
+   the :py:class:`threading.local` class::
+
+      static int
+      local_traverse(PyObject *op, visitproc visit, void *arg)
+      {
+          localobject *self = (localobject *) op;
+          Py_VISIT(Py_TYPE(self));
+          Py_VISIT(self->args);
+          Py_VISIT(self->kw);
+          Py_VISIT(self->dict);
+          return 0;
+      }
+
+   .. note::
+      :c:func:`Py_VISIT` requires the *visit* and *arg* parameters to
+      :c:func:`!local_traverse` to have these specific names; don't name them just
+      anything.
+
+   Instances of :ref:`heap-allocated types <heap-types>` hold a reference to
+   their type. Their traversal function must therefore visit the type::
+
+       Py_VISIT(Py_TYPE(self));
+
+   Alternately, the type may delegate this responsibility by
+   calling ``tp_traverse`` of a heap-allocated superclass (or another
+   heap-allocated type, if applicable).
+   If they do not, the type object may not be garbage-collected.
+
+   If the :c:macro:`Py_TPFLAGS_MANAGED_DICT` bit is set in the
+   :c:member:`~PyTypeObject.tp_flags` field, the traverse function must call
+   :c:func:`PyObject_VisitManagedDict` like this::
+
+       int err = PyObject_VisitManagedDict((PyObject*)self, visit, arg);
+       if (err) {
+           return err;
+       }
+
+   Only the members that the instance *owns* (by having
+   :term:`strong references <strong reference>` to them) must be
+   visited. For instance, if an object supports weak references via the
+   :c:member:`~PyTypeObject.tp_weaklist` slot, the pointer supporting
+   the linked list (what *tp_weaklist* points to) must **not** be
+   visited as the instance does not directly own the weak references to itself.
+
+   The traversal function has a limitation:
+
+   .. warning::
+
+      The traversal function must not have any side effects.  Implementations
+      may not modify the reference counts of any Python objects nor create or
+      destroy any Python objects, directly or indirectly.
+
+   This means that *most* Python C API functions may not be used, since
+   they can raise a new exception, return a new reference to a result object,
+   have internal logic that uses side effects.
+   Also, unless documented otherwise, functions that happen to not have side
+   effects may start having them in future versions, without warning.
+
+   For a list of safe functions, see a
+   :ref:`separate section <duringgc-functions>` below.
+
+   .. note::
+
+      The :c:func:`Py_VISIT` call may be skipped for those members that provably
+      cannot participate in reference cycles.
+      In the ``local_traverse`` example above, there is also a ``self->key``
+      member, but it can only be ``NULL`` or a Python string and therefore
+      cannot be part of a reference cycle.
+
+      On the other hand, even if you know a member can never be part of a cycle,
+      as a debugging aid you may want to visit it anyway just so the :mod:`gc`
+      module's :func:`~gc.get_referents` function will include it.
+
+   .. note::
+
+      The :c:member:`~PyTypeObject.tp_traverse` function can be called from any
+      thread.
 
-To simplify writing :c:member:`~PyTypeObject.tp_traverse` handlers, a :c:func:`Py_VISIT` macro is
-provided.  In order to use this macro, the :c:member:`~PyTypeObject.tp_traverse` implementation
-must name its arguments exactly *visit* and *arg*:
+      .. impl-detail::
 
+         Garbage collection is a "stop-the-world" operation:
+         even in :term:`free threading` builds, only one thread state is
+         :term:`attached <attached thread state>` when :c:member:`!tp_traverse`
+         handlers run.
+
+   .. versionchanged:: 3.9
+
+      Heap-allocated types are expected to visit ``Py_TYPE(self)`` in
+      ``tp_traverse``.  In earlier versions of Python, due to
+      `bug 40217 <https://bugs.python.org/issue40217>`_, doing this
+      may lead to crashes in subclasses.
+
+To simplify writing :c:member:`~PyTypeObject.tp_traverse` handlers,
+a :c:func:`Py_VISIT` macro is provided.
+In order to use this macro, the :c:member:`~PyTypeObject.tp_traverse`
+implementation must name its arguments exactly *visit* and *arg*:
 
 .. c:macro:: Py_VISIT(o)
 
-   If the :c:expr:`PyObject *` *o* is not ``NULL``, call the *visit* callback, with arguments *o*
-   and *arg*.  If *visit* returns a non-zero value, then return it.
-   Using this macro, :c:member:`~PyTypeObject.tp_traverse` handlers
-   look like::
+   If the :c:expr:`PyObject *` *o* is not ``NULL``, call the *visit*
+   callback, with arguments *o* and *arg*.
+   If *visit* returns a non-zero value, then return it.
 
-      static int
-      my_traverse(Noddy *self, visitproc visit, void *arg)
-      {
-          Py_VISIT(self->foo);
-          Py_VISIT(self->bar);
-          return 0;
-      }
+   This corresponds roughly to::
 
-The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or ``NULL``
-if the object is immutable.
+      #define Py_VISIT(o)                             \
+         if (op) {                                    \
+            int visit_result = visit(o, arg);         \
+            if (visit_result != 0) {                  \
+               return visit_result;                   \
+            }                                         \
+         }
 
 
-.. c:type:: int (*inquiry)(PyObject *self)
+Traversal-safe functions
+^^^^^^^^^^^^^^^^^^^^^^^^
 
-   Drop references that may have created reference cycles.  Immutable objects
-   do not have to define this method since they can never directly create
-   reference cycles.  Note that the object must still be valid after calling
-   this method (don't just call :c:func:`Py_DECREF` on a reference).  The
-   collector will call this method if it detects that this object is involved
-   in a reference cycle.
+The following functions and macros are safe to use in a
+:c:member:`~PyTypeObject.tp_traverse` handler:
+
+* the *visit* function passed to ``tp_traverse``
+* :c:func:`Py_VISIT`
+* :c:func:`Py_SIZE`
+* :c:func:`Py_TYPE`: if called from a :c:member:`!tp_traverse` handler,
+  :c:func:`!Py_TYPE`'s result will be valid for the duration of the handler call
+* :c:func:`PyObject_VisitManagedDict`
+* :c:func:`PyObject_TypeCheck`, :c:func:`PyType_IsSubtype`,
+  :c:func:`PyType_HasFeature`
+* :samp:`Py{<type>}_Check` and :samp:`Py{<type>}_CheckExact` -- for example,
+  :c:func:`PyTuple_Check`
+* :ref:`duringgc-functions`
+
+.. _duringgc-functions:
+
+"DuringGC" functions
+^^^^^^^^^^^^^^^^^^^^
+
+The following functions should *only* be used in a
+:c:member:`~PyTypeObject.tp_traverse` handler; calling them in other
+contexts may have unintended consequences.
+
+These functions act like their counterparts without the ``_DuringGC`` suffix,
+but they are guaranteed to not have side effects, they do not set an exception
+on failure, and they return/set :term:`borrowed references <borrowed reference>`
+as detailed in the individual documentation.
+
+Note that these functions may fail (return ``NULL`` or ``-1``),
+but as they do not set an exception, no error information is available.
+In some cases, failure is not distinguishable from a successful ``NULL`` result.
+
+.. c:function:: void *PyObject_GetTypeData_DuringGC(PyObject *o, PyTypeObject *cls)
+                void *PyObject_GetItemData_DuringGC(PyObject *o)
+                void *PyType_GetModuleState_DuringGC(PyTypeObject *type)
+                void *PyModule_GetState_DuringGC(PyObject *module)
+                int PyModule_GetToken_DuringGC(PyObject *module, void** result)
+
+   See :ref:`duringgc-functions` for common information.
+
+   .. versionadded:: next
+
+   .. seealso::
+
+      :c:func:`PyObject_GetTypeData`,
+      :c:func:`PyObject_GetItemData`,
+      :c:func:`PyType_GetModuleState`,
+      :c:func:`PyModule_GetState`,
+      :c:func:`PyModule_GetToken`,
+      :c:func:`PyType_GetBaseByToken`
+
+.. c:function:: int PyType_GetBaseByToken_DuringGC(PyTypeObject *type, void *tp_token, PyTypeObject **result)
+
+   See :ref:`duringgc-functions` for common information.
+
+   Sets *\*result* to a :term:`borrowed reference` rather than a strong one.
+   The reference is valid for the duration
+   of the :c:member:`!tp_traverse` handler call.
+
+   .. versionadded:: next
+
+   .. seealso:: :c:func:`PyType_GetBaseByToken`
+
+.. c:function:: PyObject* PyType_GetModule_DuringGC(PyTypeObject *type)
+                PyObject* PyType_GetModuleByToken_DuringGC(PyTypeObject *type, const void *mod_token)
+
+   See :ref:`duringgc-functions` for common information.
+
+   These functions return a :term:`borrowed reference`, which is
+   valid for the duration of the :c:member:`!tp_traverse` handler call.
+
+   .. versionadded:: next
+
+   .. seealso::
+
+      :c:func:`PyType_GetModule`,
+      :c:func:`PyType_GetModuleByToken`
 
 
 Controlling the Garbage Collector State

Doc/c-api/typeobj.rst

@@ -1563,93 +1563,9 @@ and :c:data:`PyType_Type` effectively act as defaults.)
    .. corresponding-type-slot:: Py_tp_traverse
 
    An optional pointer to a traversal function for the garbage collector.  This is
-   only used if the :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit is set.  The signature is::
+   only used if the :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit is set.
 
-      int tp_traverse(PyObject *self, visitproc visit, void *arg);
-
-   More information about Python's garbage collection scheme can be found
-   in section :ref:`supporting-cycle-detection`.
-
-   The :c:member:`~PyTypeObject.tp_traverse` pointer is used by the garbage collector to detect
-   reference cycles. A typical implementation of a :c:member:`~PyTypeObject.tp_traverse` function
-   simply calls :c:func:`Py_VISIT` on each of the instance's members that are Python
-   objects that the instance owns. For example, this is function :c:func:`!local_traverse` from the
-   :mod:`!_thread` extension module::
-
-      static int
-      local_traverse(PyObject *op, visitproc visit, void *arg)
-      {
-          localobject *self = (localobject *) op;
-          Py_VISIT(self->args);
-          Py_VISIT(self->kw);
-          Py_VISIT(self->dict);
-          return 0;
-      }
-
-   Note that :c:func:`Py_VISIT` is called only on those members that can participate
-   in reference cycles.  Although there is also a ``self->key`` member, it can only
-   be ``NULL`` or a Python string and therefore cannot be part of a reference cycle.
-
-   On the other hand, even if you know a member can never be part of a cycle, as a
-   debugging aid you may want to visit it anyway just so the :mod:`gc` module's
-   :func:`~gc.get_referents` function will include it.
-
-   Heap types (:c:macro:`Py_TPFLAGS_HEAPTYPE`) must visit their type with::
-
-       Py_VISIT(Py_TYPE(self));
-
-   It is only needed since Python 3.9. To support Python 3.8 and older, this
-   line must be conditional::
-
-       #if PY_VERSION_HEX >= 0x03090000
-           Py_VISIT(Py_TYPE(self));
-       #endif
-
-   If the :c:macro:`Py_TPFLAGS_MANAGED_DICT` bit is set in the
-   :c:member:`~PyTypeObject.tp_flags` field, the traverse function must call
-   :c:func:`PyObject_VisitManagedDict` like this::
-
-       PyObject_VisitManagedDict((PyObject*)self, visit, arg);
-
-   .. warning::
-       When implementing :c:member:`~PyTypeObject.tp_traverse`, only the
-       members that the instance *owns* (by having :term:`strong references
-       <strong reference>` to them) must be
-       visited. For instance, if an object supports weak references via the
-       :c:member:`~PyTypeObject.tp_weaklist` slot, the pointer supporting
-       the linked list (what *tp_weaklist* points to) must **not** be
-       visited as the instance does not directly own the weak references to itself
-       (the weakreference list is there to support the weak reference machinery,
-       but the instance has no strong reference to the elements inside it, as they
-       are allowed to be removed even if the instance is still alive).
-
-   .. warning::
-      The traversal function must not have any side effects.  It must not
-      modify the reference counts of any Python objects nor create or destroy
-      any Python objects.
-
-   Note that :c:func:`Py_VISIT` requires the *visit* and *arg* parameters to
-   :c:func:`!local_traverse` to have these specific names; don't name them just
-   anything.
-
-   Instances of :ref:`heap-allocated types <heap-types>` hold a reference to
-   their type. Their traversal function must therefore either visit
-   :c:func:`Py_TYPE(self) <Py_TYPE>`, or delegate this responsibility by
-   calling ``tp_traverse`` of another heap-allocated type (such as a
-   heap-allocated superclass).
-   If they do not, the type object may not be garbage-collected.
-
-   .. note::
-
-      The :c:member:`~PyTypeObject.tp_traverse` function can be called from any
-      thread.
-
-   .. versionchanged:: 3.9
-
-      Heap-allocated types are expected to visit ``Py_TYPE(self)`` in
-      ``tp_traverse``.  In earlier versions of Python, due to
-      `bug 40217 <https://bugs.python.org/issue40217>`_, doing this
-      may lead to crashes in subclasses.
+   See :ref:`gc-traversal` for documentation.
 
    **Inheritance:**
 

Doc/data/refcounts.dat

@@ -2430,10 +2430,17 @@ PyType_GetName:PyTypeObject*:type:0:
 PyType_GetModule:PyObject*::0:
 PyType_GetModule:PyTypeObject*:type:0:
 
+PyType_GetModule_DuringGC:PyObject*::0:
+PyType_GetModule_DuringGC:PyTypeObject*:type:0:
+
 PyType_GetModuleByToken:PyObject*::+1:
 PyType_GetModuleByToken:PyTypeObject*:type:0:
 PyType_GetModuleByToken:PyModuleDef*:def::
 
+PyType_GetModuleByToken_DuringGC:PyObject*::0:
+PyType_GetModuleByToken_DuringGC:PyTypeObject*:type:0:
+PyType_GetModuleByToken_DuringGC:PyModuleDef*:mod_token::
+
 PyType_GetModuleByDef:PyObject*::0:
 PyType_GetModuleByDef:PyTypeObject*:type:0:
 PyType_GetModuleByDef:PyModuleDef*:def::