Skip to content

feat(codegen): replace preprocess_schemas.py with ucp-schema generate-types - #101

Open
gsmith85 wants to merge 2 commits into
mainfrom
feat/ucp-schema-generate-types
Open

gsmith85 wants to merge 2 commits into
mainfrom
feat/ucp-schema-generate-types

Conversation

@gsmith85

@gsmith85 gsmith85 commented Oct 5, 2026 •

Copy link
Copy Markdown

Summary

Replaces the bespoke 373-line preprocess_schemas.py script and 140-file src/ucp_sdk/models/schemas/{shopping,common,transports}/ submodule hierarchy with on-the-fly ucp-schema generate-types compilation into a flat Pydantic v2 model module (src/ucp_sdk/models/schemas/__init__.py, re-exported from ucp_sdk.models).

Key Changes

  • generate_models.sh & CI (tests.yml):
    • Resolves ucp-schema via ${UCP_SCHEMA_BIN:-} -> command -v ucp-schema (verifying generate-types support) -> sibling ../ucp-schema/target/{release,debug}/ucp-schema -> cargo build --release --manifest-path ../ucp-schema/Cargo.toml.
    • Installs the Rust toolchain and ucp-schema CLI in the model-drift GitHub Actions job so CI model regeneration is self-contained.
    • Compiles the self-contained JSON Schema 2020-12 $defs bundle via ucp-schema generate-types --schema-dir "$SCHEMA_DIR" ($TMP_TYPES_JSON), invokes datamodel-codegen --input-file-type jsonschema --output src/ucp_sdk/models/schemas/__init__.py --use-title-as-name so inline objects, array items, and union branches use qualified semantic model names (TotalsItem, ValueConstraintEnum, ValueConstraintConst, ServiceBusinessSchemaRest, LocationServesAddressCountry, UnitPriceMeasure, etc.), and passes $TMP_TYPES_JSON directly to postprocess_models.py.
    • Removes the unused templates/pydantic_v2/RootModel.jinja2 template override.
  • Deleted Legacy Preprocessing & Submodule Tree:
    • Deleted preprocess_schemas.py and the 140-file src/ucp_sdk/models/schemas/{shopping,common,transports}/ directory tree.
    • Updated src/ucp_sdk/models/__init__.py to re-export all public symbols from ucp_sdk.models.schemas.
  • postprocess_models.py Adaptations:
    • Consumes the compiled ucp-schema generate-types $defs bundle ($TMP_TYPES_JSON) directly as its single source of truth via load_schema_defs(types_json_path) instead of re-walking raw SCHEMA_DIR files or maintaining hardcoded definition-name alias maps.
    • Scoped per-class idempotency guards for the flat single-file module layout, normalizes datamodel-code-generator's Dict[ annotations in __pydantic_extra__, and strips the synthetic UCPSchemaTypes / UcpSchemaTypes root bundle alias.
    • Injects uniqueItems constraints from $defs into generated classes (Oauth2Provider.required_claims, ValueConstraintEnum.enum, ValueConstraintConst.enum).
    • Injects canonical dependentRequired validators from $defs into generated classes (Location.timezone when hours or exception_hours is present, FulfillmentMethodUpdateRequestBase requiring type when destinations is present, PostalAddress / FulfillmentAddress street address dependencies).
    • Applies Total / TotalsItem conditional numeric bounds and custom-type display_text requirements, and Unit C62 scale-zero constraints on Measure, UnitPriceMeasure, and UnitPriceReference.
    • Injects not.required forbidden-key validators (JwkPublicKey rejecting private-key JWK fields d, p, q, dp, dq, qi, oth, k) and open-union not.enum discriminator guards on <Union>Base fallback classes (FulfillmentDestinationBase, FulfillmentMethodBase, MediaBase, ProviderBase) so payloads carrying a known discriminator tag with malformed fields raise ValidationError rather than falling back to <Union>Base.
  • Tests & Documentation:
    • Added tests/test_models.py testing known and unknown FulfillmentDestination and FulfillmentMethod variants, full Checkout payload round-tripping with unknown extension variants, missing discriminator rejection, and rejection of malformed known variants.
    • Expanded tests/test_codegen_pipeline.py with tests for semantic model names, omission of response-only request slices (OrderCreateRequest, TotalCreateRequest, TotalsCreateRequest), ShippingMethodUpdateRequest/PickupMethodUpdateRequest required type discriminator and FulfillmentMethodUpdateRequestBase typeless updates + dependentRequired: {"destinations": ["type"]}, CapabilityBase.extends scalar-or-list union (str | list[str]), ValueConstraintEnum/Oauth2Provider uniqueItems, Location timezone dependency, TotalsItem/UnitPriceMeasure conditional bounds, and JwkPublicKey forbidden private-key fields.

Stacked PR Series Roadmap (ucp-schema Two-Layer Codegen Pipeline)

Phase / PR Focus Status
PR 1a (ucp-schema#81) Codegen normalizer (to_pascal_case, strip_ucp_keywords, rewrite_refs_to_defs, canonical title normalization) Open
PR 1b (ucp-schema#82) Directional schema slicing (slice_directional_schemas, align_directional_refs) Open
PR 1c (ucp-schema#83) Capability/extension reachability, collision-aware $defs hoisting, overlay composition & generate_types() API Open
PR 1d (ucp-schema#84) ucp-schema generate-types CLI subcommand & end-to-end integration tests Open
PR 2 (ucp-schema#85) Self-contained .well-known/ucp --profile mode (src/codegen/profile.rs) Open
PR 3a (ucp-schema#86) AST shape normalizers, inline title qualification & pre-slicing inline conditional variant hoisting (src/codegen/reify.rs) Open
PR 3b (ucp-schema#87) Ordered anyOf <Union>Base open-union lowering (lower_conditional_unions) Open
PR 3c (ucp-schema#88) Decentralized extension subtype discovery (register_extended_subtypes) Open
PR 4a (this PR) Replace python-sdk preprocess_schemas.py with ucp-schema generate-types Current PR
PR 4b (js-sdk#87) Replace js-sdk quicktype pre-processors & src/extensions.ts with ucp-schema generate-types + json-schema-to-zod Open
PR 5 (ucp-schema) Layer 2 ucp-schema generate-openapi binding service.schema (rest.openapi.json) to Layer 1 types Planned

Verification

  • ./generate_models.sh ../ucp and ./generate_models.sh 2026-08-25
  • uv run python -m unittest discover -s tests -v (150 tests passed)
  • uv run ruff check . && uv run ruff format --check .

@damaz91 damaz91 added the status:needs-triage Signal that the PR is ready for human triage label Oct 5, 2026
@gsmith85
gsmith85 force-pushed the feat/ucp-schema-generate-types branch from fac80be to 0e3dee5 Compare October 6, 2026 00:14
@damaz91 damaz91 added status:under-review and removed status:needs-triage Signal that the PR is ready for human triage labels Oct 6, 2026
],
)
"""
Key-value map whose keys represent buyer/platform asserted eligibility claims and whose values represent associated membership information. All loyalty keys MUST use reverse-domain naming to ensure provenance and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ap2 is generated as a required field on CheckoutCompleteRequest, which breaks validation for any non-AP2 checkout completion request.

In common/payment_ap2_mandate.json, $defs["dev.ucp.shopping.checkout"].properties.ap2 sets "ucp_request": {"create": "omit", "update": "omit", "complete": "required"}. Because ucp-schema generate-types flattens optional capability extensions onto the unified CheckoutCompleteRequest model, top-level extension properties contributed by overlays need to remain optional (ap2: Ap2WithCheckoutMandateCompleteRequest | None = None), while checkout_mandate stays required inside Ap2WithCheckoutMandateCompleteRequest when ap2 is present.

Comment on lines +7924 to +7936
"""
Errors, warnings, or informational messages about the search results.
"""


Loyalty = TypeAliasType(
"Loyalty",
Annotated[
dict[ReverseDomainName, LoyaltyMembership], Field(..., title="Loyalty")
],
)
"""
Key-value map whose keys represent buyer/platform asserted eligibility claims and whose values represent associated membership information. All loyalty keys MUST use reverse-domain naming to ensure provenance and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

buyer and cart_id should be omitted from CheckoutCompleteRequest:

  • In shopping/cart.json, cart_id defines "ucp_request": {"create": "optional", "update": "omit"} (omitting "complete").
  • In shopping/buyer_consent.json, buyer defines "ucp_request": {"create": "optional", "update": "optional"} (omitting "complete", while base shopping/checkout.json sets "complete": "omit").

When ucp_request is an operation map, operations not listed in the map should default to "omit" rather than "optional".

Comment on lines +7949 to +7966
"""
Locations matching the search criteria.
"""
pagination: PaginationResponse | None = None
messages: list[Message] | None = None
"""
Errors, warnings, or informational messages about the search results.
"""


Loyalty = TypeAliasType(
"Loyalty",
Annotated[
dict[ReverseDomainName, LoyaltyMembership], Field(..., title="Loyalty")
],
)
"""
Key-value map whose keys represent buyer/platform asserted eligibility claims and whose values represent associated membership information. All loyalty keys MUST use reverse-domain naming to ensure provenance and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making line_items: list[LineItemCreateRequest] unconditionally required on the unified CheckoutCreateRequest prevents cart-to-checkout conversion using only cart_id (per shopping/cart.json: "Business MUST use cart contents (line_items, context, buyer) and MUST ignore overlapping fields in checkout payload").

Previously, line_items was optional (list[LineItemCreateRequest] | None = None) when cart_id was in scope, and postprocess_models.py injected _enforce_cart_conversion to require at least one of cart_id or line_items.

@gsmith85
gsmith85 force-pushed the feat/ucp-schema-generate-types branch from 0e3dee5 to 2c1cec5 Compare October 6, 2026 17:23

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants