PEP 825: Various clarifications (#4969)

Author: konstin

peps/pep-0825.rst

@@ -308,11 +308,17 @@ there MUST exist a corresponding ``{name}-{version}-variants.json``
 file. The ``{name}`` and ``{version}`` placeholders correspond to the
 package name and version, normalized according to the same rules as
 wheel files, as found in the :ref:`packaging:wheel-file-name-spec` of
-the Binary Distribution Format specification. The exact URL where the
-file is hosted is insignificant, but a link to it MUST be present on all
-index pages where the variant wheels are linked. It is presented in the
-same simple repository format as source distribution and wheel links in
-the index, including an (OPTIONAL) hash.
+the Binary Distribution Format specification.
+
+The exact URL where the file is hosted is insignificant, but it MUST
+be provided in all the responses where the variant wheels are included.
+It should follow the rules for files in the
+:ref:`packaging:simple-repository-api`, except that the optional
+metadata attributes served by the index (such as ``core-metadata``,
+``dist-info-metadata``, ``requires-python`` or ``yanked``) are not
+meaningful for that file. Indexes MAY publish or skip these attributes,
+as long as the values do not prevent correct operation. Tools MAY either
+use or ignore these values.
 
 This file uses the same structure as `variant metadata`_, except that
 the ``variants`` object MUST list all variants available on the package
@@ -379,17 +385,20 @@ like:
 Variant ordering
 ----------------
 
-To determine which variant wheel to install when multiple wheels are
-compatible, variants MUST be totally ordered by their variant
-properties.
+This specification defines an ordering between different wheels based on
+the presence of variant metadata.
 
 For the purpose of ordering, variant properties are grouped into
 features, and features into namespaces. For every namespace, the tool
-MUST obtain an ordered list of compatible features, and for every
-feature, an ordered list of compatible values. The method of obtaining
-these lists will be defined in a subsequent PEP.
-
-The default ordering MUST be performed equivalent to the following
+MUST obtain a list of compatible features, and for every feature, a list
+of compatible values. The method of obtaining these lists will be
+defined in a subsequent PEP. The items in these lists will be provided
+in specific order that will impact variant wheel ordering.
+
+The compatible wheels corresponding to a particular combination of
+package name, version and build number MUST be grouped by their variant
+label, and a separate group of non-variant wheels MUST be formed. The
+groups of variant wheels MUST then be ordered according to the following
 algorithm:
 
 1. Construct the ordered list of namespaces by copying the value of the
@@ -402,9 +411,10 @@ algorithm:
       value of the respective ``default-priorities.feature.{namespace}``
       key.
 
-   ii. Obtain the compatible feature names, in order. For every feature
-       name that is not present in the constructed list, append it to
-       the end.
+   ii. Take the ordered list of compatible feature names obtained
+       previously and iterate over it, in order. For every feature name
+       that is not present in the constructed list, append it to the
+       end.
 
    After this step, a list of ordered feature names is available for
    every namespace. This is ``feature_order`` in the example.
@@ -415,48 +425,49 @@ algorithm:
       of the respective
       ``default-priorities.property.{namespace}.{feature_name}`` key.
 
-   ii. Obtain the compatible feature values, in order. For every value
-       that is not present in the constructed list, append it to the
-       end.
+   ii. Take the ordered list of compatible feature values obtained
+       previously and iterate over it, in order. For every value that is
+       not present in the constructed list, append it to the end.
 
    After this step, a list of ordered property values is available for
    every feature. This is ``value_order`` in the example.
 
-4. For every compatible variant, determine the most preferred value
-   corresponding to every feature in that variant. This is done by
-   finding among the values present in the variant properties the one
-   that has the lowest position in the ordered property value list.
-   After this step, a list of features along with their best values
-   is available for every variant. This is done in the
-   ``Variant.best_value_properties()`` method in the example.
+4. For every group, determine the most preferred value corresponding to
+   every variant feature present in the variant properties corresponding
+   to the group. This is done by finding among the values the one that
+   has the lowest position in the ordered property value list. After
+   this step, a list of features along with their best values is
+   available for every variant. This is done in the
+   ``VariantWheel.best_value_properties()`` method in the example.
 
 5. For every item in the list constructed in the previous step,
    construct a sort key that is a 3-tuple consisting of
    its namespace, feature name and best feature value indices in the
    respective ordered lists. This is done by the ``property_key()``
    function in the example.
 
-6. For every compatible variant, sort the list constructed in step 4
-   using the sort keys constructed in step 5, in ascending order. This
-   is done by the ``Variant.sorted_properties()`` method in the example.
+6. For every group, sort the list constructed in step 4 using the sort
+   keys constructed in step 5, in ascending order. This is done by the
+   ``VariantWheel.sorted_properties()`` method in the example.
 
-7. To order variants, compare their sorted lists from step 6. If the
-   sort keys at the first position are different, the variant with the
+7. To order groups, compare their sorted lists from step 6. If the
+   sort keys at the first position are different, the group with the
    lower key is sorted earlier. If they are the same, compare the keys
    at the second position, and so on, until either a tie-breaker is
-   found or the list in one of the variants is exhausted. In the latter
-   case, the variant with more keys is sorted earlier. As a fallback,
-   if both variants have the same number of keys, they are ordered
-   lexically by their variant label, ascending. This is done by the
+   found or the list in one of the groups is exhausted. In the latter
+   case, the group with more keys is sorted earlier. As a fallback,
+   if both groups have the same number of keys, they are ordered
+   lexically by the variant label, ascending. This is done by the
    ultimate step of the example algorithm, with the comparison function
-   being implemented as ``Variant.__lt__()``.
+   being implemented as ``VariantWheel.__lt__()``.
 
-After this process, the variant wheels are sorted from the most
-preferred to the least preferred. The algorithm sorts the null variant
-after all the other variants. The non-variant wheel MUST be ordered
-after the null variant. Multiple wheels with the same variant property
-set (and multiple non-variant wheels) MUST then be ordered according to
-their platform compatibility tags.
+The algorithm sorts the group of null variant wheels last, as they
+feature no variant properties. The group of non-variant wheels MUST be
+placed after all the other groups.
+
+Within every group, the wheels MUST then be ordered according to their
+platform compatibility tags. After this process, the variant wheels are
+sorted from the most preferred to the least preferred.
 
 The tools MAY provide options to override the default ordering, for
 example by specifying a preference for specific namespaces, features
@@ -465,7 +476,8 @@ variants, or to select a particular variant.
 
 Alternatively, the sort algorithm for variant wheels could be described
 using the following pseudocode. For simplicity, this code does not
-account for non-variant wheels or tags.
+account for non-variant wheels or the subsequent ordering by platform
+compatibility tags.
 
 .. code:: python
 
@@ -530,7 +542,7 @@ account for non-variant wheels or tags.
         )
 
 
-    class Variant:
+    class VariantWheel:
         """Example class exposing properties of a variant wheel"""
 
         label: str
@@ -571,13 +583,13 @@ account for non-variant wheels or tags.
             return self.label < other.label
 
 
-    # A list of variants to sort.
-    variants: list[Variant] = [...]
+    # A list of variant wheels to sort.
+    variant_wheels: list[VariantWheel] = [...]
 
 
-    # 7. Order variants by comparing their sorted properties
-    # (see Variant.__lt__())
-    variants.sort()
+    # 7. Order variant wheels by comparing their sorted properties
+    # (see VariantWheel.__lt__())
+    variant_wheels.sort()
 
 
 Environment markers
@@ -737,6 +749,10 @@ Note that steps 4. through 8. are introduced specifically for variant
 wheels. The remaining steps correspond to the current installer
 behavior.
 
+When installing from a source that does not provide an `index-level
+metadata`_, the same algorithm can be used, except that the variant
+metadata needs to be read directly from the wheels.
+
 
 Installing a local wheel
 ''''''''''''''''''''''''
@@ -786,28 +802,68 @@ To generate the ``{name}-{version}-variants.json`` file:
 Rationale
 =========
 
+This PEP is part of a larger variant wheel design that was originally
+proposed as :pep:`817`. However, due to its complexity, we decided to
+split it into smaller parts that build one upon another. This PEP is the
+first in the series, providing foundations including the file format
+along with necessary metadata, index support and basic tool algorithms.
+Aspects such as providing actual variant properties or building wheels
+are deferred into subsequent PEPs.
+
 Variant wheels use structured `variant properties`_ to express
-multidimensional wheel compatibility matrices. For example, it permits
-expressing that a single variant requires certain CPU and GPU features
-independently. It can express both AND-style dependencies (such as
-different CPU instruction sets) and OR-style dependencies (such as
-different GPUs supported by a single package).
+multidimensional wheel compatibility matrices. Properties are organized
+in namespaces that can be defined and governed independently. The
+key-value structure makes the properties more flexible: adding a new
+compatibility axis can be done by adding a new key. It can support both
+AND-style dependencies (for example, a CPU plugin could define multiple
+keys corresponding to different instructions sets, all of which are used
+in the package and therefore must be supported) and OR-style
+dependencies (for example, a GPU plugin can define a single key listing
+multiple GPU types, indicating that all of them are supported by the
+package, and therefore the users needs to own only one of them).
 
 The specification does not impose any formal limits on the number of
 properties expressed, and specifically accounts for the possibility of
 property sets being very long (for example, a long list of GPUs or CPU
-extension sets). To avoid wheel filenames becoming very long, the
-property lists are stored inside the wheel and mapped to a short label
-that is intended to be human-readable.
-
-To facilitate variant selection while installing from remote index,
-the variant metadata is mirrored in a JSON file published on the index.
-This enables installers to obtain variant property mapping without
-having to fetch individual wheels.
-
-Since JSON format does not feature a set type, sets are represented as
-sorted lists. Sorting ensures that tools can safely use equality
-comparison over dictionaries.
+extension sets). To avoid wheel filenames becoming hard to comprehend
+because of excess of information and potentially causing technical
+issues because of their length, the property lists are stored inside
+the wheel and mapped to a short label that is chosen by the package
+maintainer and intended to be human-readable.
+
+Wheel filenames alone do not provide sufficient metadata to drive
+variant wheel selection. To avoid tools having to fetch the variant
+metadata straight from multiple wheel files, the metadata from wheels
+for every package version is combined and republished. This metadata is
+scoped to a single package version to permit variants changing in the
+future version.
+
+The index support aims to account for three scenarios:
+
+1. An index implementation that cannot embed additional metadata as part
+   of file list responses. For example, this covers installing straight
+   from a directory listing created by a webserver. To account for this
+   scenario, index-level metadata is published as a plain JSON file that
+   can be generated by the package maintainer and placed alongside
+   wheels.
+
+2. An index implementation that has more complete wheel support but does
+   not wish to implement full variant wheel support immediately. The
+   index needs only to permit the user to upload said JSON file. To
+   account for minimalistic implementation, the specification permits
+   the index to treat said file similarly to a wheel, including
+   publishing attributes such as ``yanked``, as long as their values do
+   not prevent clients from working.
+
+3. An index implementation that implements complete wheel variant
+   support. Such an index will parse uploaded variant wheels, and
+   dynamically create the index-level metadata. The JSON file path would
+   then be treated as an API endpoint rather than an actual file.
+
+Since JSON format does not feature a set type, sets in the metadata are
+represented as sorted lists. Sorting ensures reproducibility and makes
+it possible to use equality comparison over whole dictionaries without
+having to convert specific fields back to sets after deserialization.
 
 The variant ordering algorithm has been proposed with the assumption
 that variant properties take precedence over Platform compatibility
@@ -817,15 +873,25 @@ variant may require a different minimal libc version, in which case the
 selection should be driven by the desired CUDA preference rather than
 incidental platform tag difference.
 
-While the provision of variant properties is deferred to a future PEP to
-keep the specification easier to comprehend, a baseline assumption is
-made that the compatible properties will be provided in specific order
-corresponding to their preference, much like Platform compatibility tags
-conventionally are. The variant metadata provides the ability to
-override this order at package level. However, namespaces are unordered
-by design (e.g. we will not decide upfront which GPU vendors take
-precedence) and therefore they always need to be ordered by the package
-maintainer.
+While a future PEP will define how variant properties are provided, a
+baseline assumption is made that the compatible properties will be
+provided in specific order corresponding to their preference. This makes
+it possible to use a generic sorting algorithm, and later define
+properties as data without having to change the algorithm.
+
+A future PEP will define how the ordering for features and values is
+provided. However, namespaces are governed independently and considered
+on equal footing, and therefore there will be no standard ordering for
+them. Instead, the ordering of namespaces will be explicitly stated in
+the variants metadata, which in turn will be provided by the package
+maintainer as part of the build process. For completeness, it will also
+be possible to provide overrides for the ordering of features and values
+via the same mechanism.
+
+In the vast majority of real use cases, ordering based on properties
+will suffice. However, in a pathological case two different variant
+wheels may end up with equal sort keys. To provide reproducible results
+in this case, fallback sorting on variant label is performed.
 
 A concept of null variant is introduced that is distinct from
 non-variant wheels to facilitate a transition period. This variant is
@@ -938,7 +1004,7 @@ Reference Implementation
 The `variantlib <https://github.com/wheelnext/variantlib>`__ project
 contains a reference implementation of a complete variant wheel
 solution. It is compliant with this PEP, but also goes beyond it,
-providing example solutions to `open issues`_.
+providing example solutions to some of the deferred items.
 
 A client for installing variant wheels is implemented in a
 `uv branch <https://github.com/astral-sh/uv/pull/12203>`__.
@@ -982,15 +1048,36 @@ would be incorrectly deemed compatible because of the
 ``manylinux_2_27_x86_64`` part.
 
 
-Open Issues
-===========
+Replacing Platform compatibility tags entirely
+----------------------------------------------
+
+Technically, it would be entirely possible to convey the information
+currently passed via the Platform compatibility tags via variant
+properties, and remove these explicit tags from the filename. However,
+we decided not to pursue this and instead preserve the existing
+filenames for wheels that do not need additional variants, as we do not
+believe that the effort required to update all the existing workflows
+justifies the benefit of more compact, and slightly more consistent
+naming.
+
+
+Out of scope
+------------
 
-The following problems are deferred to subsequent PEPs:
+The following problems are deferred to subsequent PEPs in the series:
 
 - governance of variant namespaces
 - determining which variant properties are compatible with the system
+- overriding the compatibility detection using static data
 - building variant wheels
 
+In addition to that, the following matters are left
+implementation-defined:
+
+- Selecting variant wheels from multiple sources. Currently, there is no
+  standard defined behavior for regular wheels, nor consensus across
+  different packaging tools on how to handle that.
+
 
 Acknowledgements
 ================
@@ -1036,6 +1123,12 @@ Change History
   - Changed ``pylock.toml`` integration to inline variant metadata
     rather than storing a URL and a hash.
 
+- 11-May-2026
+
+  - Added replacing platform compatibility tags entirely to rejected
+    ideas.
+  - Clarified interpretation of sorting algorithm and index support.
+
 
 Appendices
 ==========