Merge branch 'main' into to_bool_specialization

Author: eendebakpt

.github/workflows/reusable-san.yml

@@ -61,7 +61,7 @@ jobs:
               || ''
             }}.txt handle_segv=0" >> "$GITHUB_ENV"
         else
-          echo "UBSAN_OPTIONS=${SAN_LOG_OPTION}" >> "$GITHUB_ENV"
+          echo "UBSAN_OPTIONS=${SAN_LOG_OPTION} halt_on_error=1 suppressions=${GITHUB_WORKSPACE}/Tools/ubsan/suppressions.txt" >> "$GITHUB_ENV"
         fi
         echo "CC=clang" >> "$GITHUB_ENV"
         echo "CXX=clang++" >> "$GITHUB_ENV"
@@ -75,18 +75,21 @@ jobs:
         ${{
           inputs.sanitizer == 'TSan'
           && '--with-thread-sanitizer'
-          || '--with-undefined-behavior-sanitizer'
+          || '--with-undefined-behavior-sanitizer --with-strict-overflow'
         }}
         --with-pydebug
         ${{ fromJSON(inputs.free-threading) && '--disable-gil' || '' }}
     - name: Build CPython
       run: make -j4
     - name: Display build info
       run: make pythoninfo
+    # test_{capi,faulthandler} are skipped under UBSan because
+    # they raise signals that UBSan with halt_on_error=1 intercepts.
     - name: Tests
       run: >-
         ./python -m test
         ${{ inputs.sanitizer == 'TSan' && '--tsan' || '' }}
+        ${{ inputs.sanitizer == 'UBSan' && '-x test_capi -x test_faulthandler' || '' }}
         -j4
     - name: Parallel tests
       if: >-

.pre-commit-config.yaml

@@ -85,6 +85,9 @@ repos:
         exclude: Lib/test/tokenizedata/coding20731.py
       - id: end-of-file-fixer
         files: '^\.github/CODEOWNERS$'
+      - id: mixed-line-ending
+        args: [--fix=auto]
+        exclude: '^Lib/test/.*data/'
       - id: trailing-whitespace
         types_or: [c, inc, python, rst, yaml]
       - id: trailing-whitespace

Doc/c-api/dict.rst

@@ -76,6 +76,11 @@ Dictionary objects
 
    The first argument can be a :class:`dict` or a :class:`frozendict`.
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
+
    .. versionchanged:: 3.15
       Also accept :class:`frozendict`.
 
@@ -105,6 +110,11 @@ Dictionary objects
    ``0`` on success or ``-1`` on failure.  This function *does not* steal a
    reference to *val*.
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
+
 
 .. c:function:: int PyDict_SetItemString(PyObject *p, const char *key, PyObject *val)
 
@@ -120,6 +130,11 @@ Dictionary objects
    If *key* is not in the dictionary, :exc:`KeyError` is raised.
    Return ``0`` on success or ``-1`` on failure.
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
+
 
 .. c:function:: int PyDict_DelItemString(PyObject *p, const char *key)
 
@@ -140,6 +155,11 @@ Dictionary objects
 
    The first argument can be a :class:`dict` or a :class:`frozendict`.
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
+
    .. versionadded:: 3.13
 
    .. versionchanged:: 3.15
@@ -162,6 +182,13 @@ Dictionary objects
       :meth:`~object.__eq__` methods are silently ignored.
       Prefer the :c:func:`PyDict_GetItemWithError` function instead.
 
+   .. note::
+
+      In the :term:`free-threaded build`, the returned
+      :term:`borrowed reference` may become invalid if another thread modifies
+      the dictionary concurrently. Prefer :c:func:`PyDict_GetItemRef`, which
+      returns a :term:`strong reference`.
+
    .. versionchanged:: 3.10
       Calling this API without an :term:`attached thread state` had been allowed for historical
       reason. It is no longer allowed.
@@ -177,6 +204,13 @@ Dictionary objects
    occurred.  Return ``NULL`` **without** an exception set if the key
    wasn't present.
 
+   .. note::
+
+      In the :term:`free-threaded build`, the returned
+      :term:`borrowed reference` may become invalid if another thread modifies
+      the dictionary concurrently. Prefer :c:func:`PyDict_GetItemRef`, which
+      returns a :term:`strong reference`.
+
    .. versionchanged:: 3.15
       Also accept :class:`frozendict`.
 
@@ -195,6 +229,13 @@ Dictionary objects
       Prefer using the :c:func:`PyDict_GetItemWithError` function with your own
       :c:func:`PyUnicode_FromString` *key* instead.
 
+   .. note::
+
+      In the :term:`free-threaded build`, the returned
+      :term:`borrowed reference` may become invalid if another thread modifies
+      the dictionary concurrently. Prefer :c:func:`PyDict_GetItemStringRef`,
+      which returns a :term:`strong reference`.
+
    .. versionchanged:: 3.15
       Also accept :class:`frozendict`.
 
@@ -221,6 +262,14 @@ Dictionary objects
 
    .. versionadded:: 3.4
 
+   .. note::
+
+      In the :term:`free-threaded build`, the returned
+      :term:`borrowed reference` may become invalid if another thread modifies
+      the dictionary concurrently. Prefer :c:func:`PyDict_SetDefaultRef`,
+      which returns a :term:`strong reference`.
+
+
 
 .. c:function:: int PyDict_SetDefaultRef(PyObject *p, PyObject *key, PyObject *default_value, PyObject **result)
 
@@ -240,6 +289,11 @@ Dictionary objects
    These may refer to the same object: in that case you hold two separate
    references to it.
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
+
    .. versionadded:: 3.13
 
 
@@ -257,6 +311,11 @@ Dictionary objects
    Similar to :meth:`dict.pop`, but without the default value and
    not raising :exc:`KeyError` if the key is missing.
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
+
    .. versionadded:: 3.13
 
 
@@ -403,6 +462,13 @@ Dictionary objects
    only be added if there is not a matching key in *a*. Return ``0`` on
    success or ``-1`` if an exception was raised.
 
+   .. note::
+
+      In the :term:`free-threaded build`, when *b* is a
+      :class:`dict` (with the standard iterator), both *a* and *b* are locked
+      for the duration of the operation. When *b* is a non-dict mapping, only
+      *a* is locked; *b* may be concurrently modified by another thread.
+
 
 .. c:function:: int PyDict_Update(PyObject *a, PyObject *b)
 
@@ -412,6 +478,13 @@ Dictionary objects
    argument has no "keys" attribute.  Return ``0`` on success or ``-1`` if an
    exception was raised.
 
+   .. note::
+
+      In the :term:`free-threaded build`, when *b* is a
+      :class:`dict` (with the standard iterator), both *a* and *b* are locked
+      for the duration of the operation. When *b* is a non-dict mapping, only
+      *a* is locked; *b* may be concurrently modified by another thread.
+
 
 .. c:function:: int PyDict_MergeFromSeq2(PyObject *a, PyObject *seq2, int override)
 
@@ -427,13 +500,27 @@ Dictionary objects
               if override or key not in a:
                   a[key] = value
 
+   .. note::
+
+      In the :term:`free-threaded <free threading>` build, only *a* is locked.
+      The iteration over *seq2* is not synchronized; *seq2* may be concurrently
+      modified by another thread.
+
+
 .. c:function:: int PyDict_AddWatcher(PyDict_WatchCallback callback)
 
    Register *callback* as a dictionary watcher. Return a non-negative integer
    id which must be passed to future calls to :c:func:`PyDict_Watch`. In case
    of error (e.g. no more watcher IDs available), return ``-1`` and set an
    exception.
 
+   .. note::
+
+      This function is not internally synchronized. In the
+      :term:`free-threaded <free threading>` build, callers should ensure no
+      concurrent calls to :c:func:`PyDict_AddWatcher` or
+      :c:func:`PyDict_ClearWatcher` are in progress.
+
    .. versionadded:: 3.12
 
 .. c:function:: int PyDict_ClearWatcher(int watcher_id)
@@ -442,6 +529,13 @@ Dictionary objects
    :c:func:`PyDict_AddWatcher`. Return ``0`` on success, ``-1`` on error (e.g.
    if the given *watcher_id* was never registered.)
 
+   .. note::
+
+      This function is not internally synchronized. In the
+      :term:`free-threaded <free threading>` build, callers should ensure no
+      concurrent calls to :c:func:`PyDict_AddWatcher` or
+      :c:func:`PyDict_ClearWatcher` are in progress.
+
    .. versionadded:: 3.12
 
 .. c:function:: int PyDict_Watch(int watcher_id, PyObject *dict)

Doc/c-api/set.rst

@@ -89,6 +89,11 @@ the constructor functions work with any iterable Python object.
    actually iterable.  The constructor is also useful for copying a set
    (``c=set(s)``).
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *iterable* is a :class:`set`, :class:`frozenset`, :class:`dict` or :class:`frozendict`.
+
 
 .. c:function:: PyObject* PyFrozenSet_New(PyObject *iterable)
 
@@ -97,6 +102,11 @@ the constructor functions work with any iterable Python object.
    set on success or ``NULL`` on failure.  Raise :exc:`TypeError` if *iterable* is
    not actually iterable.
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *iterable* is a :class:`set`, :class:`frozenset`, :class:`dict` or :class:`frozendict`.
+
 
 The following functions and macros are available for instances of :class:`set`
 or :class:`frozenset` or instances of their subtypes.
@@ -124,6 +134,10 @@ or :class:`frozenset` or instances of their subtypes.
    the *key* is unhashable. Raise :exc:`SystemError` if *anyset* is not a
    :class:`set`, :class:`frozenset`, or an instance of a subtype.
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
 
 .. c:function:: int PySet_Add(PyObject *set, PyObject *key)
 
@@ -135,6 +149,12 @@ or :class:`frozenset` or instances of their subtypes.
    :exc:`SystemError` if *set* is not an instance of :class:`set` or its
    subtype.
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
+
+
 
 The following functions are available for instances of :class:`set` or its
 subtypes but not for instances of :class:`frozenset` or its subtypes.
@@ -149,6 +169,11 @@ subtypes but not for instances of :class:`frozenset` or its subtypes.
    temporary frozensets. Raise :exc:`SystemError` if *set* is not an
    instance of :class:`set` or its subtype.
 
+   .. note::
+
+      The operation is atomic on :term:`free threading <free-threaded build>`
+      when *key* is :class:`str`, :class:`int`, :class:`float`, :class:`bool` or :class:`bytes`.
+
 
 .. c:function:: PyObject* PySet_Pop(PyObject *set)
 
@@ -164,6 +189,12 @@ subtypes but not for instances of :class:`frozenset` or its subtypes.
    success. Return ``-1`` and raise :exc:`SystemError` if *set* is not an instance of
    :class:`set` or its subtype.
 
+   .. note::
+
+      In the :term:`free-threaded build`, the set is emptied before its entries
+      are cleared, so other threads will observe an empty set rather than
+      intermediate states.
+
 
 Deprecated API
 ^^^^^^^^^^^^^^

Doc/c-api/unicode.rst

@@ -1881,6 +1881,10 @@ object.
    On success, return ``0``.
    On error, set an exception, leave the writer unchanged, and return ``-1``.
 
+   To write a :class:`str` subclass which overrides the :meth:`~object.__str__`
+   method, :c:func:`PyUnicode_FromObject` can be used to get the original
+   string.
+
 .. c:function:: int PyUnicodeWriter_WriteRepr(PyUnicodeWriter *writer, PyObject *obj)
 
    Call :c:func:`PyObject_Repr` on *obj* and write the output into *writer*.