feat(codegen): add identifier naming, UCP keyword stripping, and $defs ref rewriting by gsmith85 · Pull Request #81 · Universal-Commerce-Protocol/ucp-schema

gsmith85 · 2026-09-30T16:10:29Z

Summary

Implements Task 1 (PR 1a) of the two-layer ucp-schema code-generation pipeline: foundational naming qualification, UCP keyword stripping, and #/$defs/ reference rewriting utilities in src/codegen/normalizer.rs.

Changes

Promote collect_schema_files from src/linter.rs to pub(crate) in src/loader.rs (excluding *.openapi.json and *.types.json bundle files) so linter and codegen share a single directory walker.
Add PascalCase and $defs qualification helpers (to_pascal_case, is_reverse_domain_name, is_generic_def_name, qualify_def_name, qualify_container_op_name, and ref_to_def_name).
Add strip_ucp_keywords to strip UCP authoring annotations (ucp_request, ucp_response, ucp_shared_request, x-ucp-*, root name/version/requires/embedded, $schema, $id) and prune dangling required entries while preserving instance-data payloads (const, enum, default, examples).
Add rewrite_refs_to_defs to rewrite self (#), local (#/$defs/*), and cross-file $ref pointers into the unified #/$defs/<TypeName> namespace.

Stacked PR Series Roadmap

PR 1a (this PR): Identifier qualification, UCP keyword stripping, $ref rewriting (src/codegen/normalizer.rs)
PR 1b (feat(codegen): add directional schema slicing and $ref alignment #82): Directional schema slicing & $ref alignment (src/codegen/normalizer.rs)
PR 1c (feat(codegen): add capability reachability, composition, and generate_types() API #83): Capability/extension reachability, composition & generate_types() library API (src/codegen/{mod,reachability,hoist,compose}.rs)
PR 1d (feat(cli): add generate-types subcommand #84): CLI generate-types subcommand (src/bin/ucp-schema.rs, tests/generate_types_test.rs)
PR 2 (feat(codegen): add self-contained .well-known/ucp profile scoping for generate-types #85): Self-contained .well-known/ucp --profile scoping (src/codegen/profile.rs)
PR 3a (feat(codegen): add AST shape normalizers and inline conditional variant hoisting #86): Codegen AST shape normalizers & inline conditional variant hoisting (src/codegen/normalizer.rs, src/codegen/reify.rs)
PR 3b (feat(codegen): lower open string discriminator unions into ordered anyOf with <Union>Base #87): Open string discriminator union lowering (<Union>Base) (src/codegen/reify.rs)
PR 3c (feat(codegen): discover and register decentralized extended subtypes into open unions #88): Cross-file extended subtype discovery & registration (src/codegen/reify.rs)
PR 4a (python-sdk#101): Python SDK direct JSON Schema codegen (generate_models.sh, postprocess_models.py, src/ucp_sdk/models/)
PR 4b (js-sdk#87): JS/TS SDK direct JSON Schema codegen (generate_models.sh, scripts/generate-zod-from-types.mjs, src/spec_generated.ts)
PR 5: OpenAPI 3.1 REST service & type binding (ucp-schema generate-openapi, src/openapi/mod.rs)

Test Plan

cargo fmt --check
cargo test --all-targets
cargo clippy --all-targets -- -D warnings

…s ref rewriting First PR in a stacked series introducing a two-layer code-generation pipeline to `ucp-schema` (replacing monolithic PR #73) so downstream SDKs (`python-sdk`, `js-sdk`) and service stub generators can consume compiled UCP schemas directly without custom preprocessing scripts: 1. [This PR] Foundational `codegen::normalizer` naming, UCP keyword stripping, and `#/$defs/` reference rewriting utilities. 2. Directional slicing (`CreateRequest`, `UpdateRequest`, `CompleteRequest`, Response) and directional `$ref` alignment. 3. Capability/extension reachability, `$defs` hoisting, extension composition, and the `generate_types()` library API. 4. `ucp-schema generate-types` CLI subcommand (`--capability`, `--extension`, `--schema-dir`) and self-contained `--profile` input. 5. Polymorphic union reification (`<Union>Base` ordered `anyOf` fallback per ucp#800 §2.4, inline variant hoisting, and subtype discovery). 6. `ucp-schema generate-openapi` CLI & library binding a profile's REST service schema (`rest.openapi.json`) to compiled capability types. Changes in this PR: - Promote `collect_schema_files` from `src/linter.rs` to `pub(crate)` in `src/loader.rs` (excluding `*.openapi.json` and `*.types.json` bundle files) so `linter` and `codegen` share a single directory walker. - Add PascalCase and `$defs` qualification helpers (`to_pascal_case`, `is_reverse_domain_name`, `is_generic_def_name`, `qualify_def_name`, `qualify_container_op_name`, and `ref_to_def_name`). - Add `strip_ucp_keywords` to strip UCP authoring annotations (`ucp_request`, `ucp_response`, `ucp_shared_request`, `x-ucp-*`, root `name`/`version`/ `requires`/`embedded`, `$schema`, `$id`) and prune dangling `required` entries while preserving instance-data payloads (`const`, `enum`, `default`, `examples`). - Add `rewrite_refs_to_defs` to rewrite self (`#`), local (`#/$defs/*`), and cross-file `$ref` pointers into the unified `#/$defs/<TypeName>` namespace.

damaz91

Thanks for splitting PR #73 into a clean, reviewable stack, @gsmith85! Left a few inline comments on collect_schema_files (interaction with ucp-schema lint), prune_dangling_required on nested anyOf/oneOf/then branches (e.g., in service.json), and rewrite_refs_to_defs / to_pascal_case idempotency.

damaz91 · 2026-10-01T13:43:06Z

+fn is_schema_json_file(path: &Path) -> bool {
+    if path.extension().and_then(|e| e.to_str()) != Some("json") {
+        return false;
+    }
+    let Some(name) = path.file_name().and_then(|n| n.to_str()) else {
+        return false;
+    };
+    !name.ends_with(".openapi.json") && !name.ends_with(".types.json")
+}
+
+/// Collect all `.json` schema files in a path (file or directory), excluding
+/// `*.openapi.json` and `*.types.json` bundle files.
+pub(crate) fn collect_schema_files(path: &Path) -> Vec<PathBuf> {
+    if path.is_file() {
+        if is_schema_json_file(path) {
+            return vec![path.to_path_buf()];
+        }
+        return vec![];
+    }
+
+    let mut files = Vec::new();
+    collect_files_recursive(path, &mut files);
+    files.sort();
+    files
+}


Bug (ucp-schema lint regression on *.openapi.json):

Because collect_schema_files is shared with linter::lint, filtering out *.openapi.json (and *.types.json) here affects the linter in two ways:

Silent CI regression in ucp: Universal-Commerce-Protocol/ucp runs ucp-schema lint source/ in CI, and src/linter.rs (is_service_definition, added in fix(linter): skip the $id check for OpenAPI and OpenRPC service definitions #49) explicitly supports linting OpenAPI (*.openapi.json, openapi.json) and OpenRPC (*.openrpc.json) service definitions for JSON syntax and $ref resolution. With this filter, ucp-schema lint source/ now silently skips source/services/common/rest.openapi.json, source/services/shopping/rest.openapi.json, and source/services/shopping/permalink.openapi.json (while still linting source/handlers/tokenization/openapi.json and *.openrpc.json).

Single-file lint becomes a no-op: Because if path.is_file() also gates on is_schema_json_file(path), running ucp-schema lint path/to/rest.openapi.json directly returns vec![] (✓ 0 files checked, all passed, exit code 0) even if the file contains broken $refs or invalid JSON.

Suggestion: Keep linter::lint collecting all .json files (and never filter out an explicitly passed file when path.is_file()), and scope the bundle/service-file exclusion (*.openapi.json, *.types.json, *.openrpc.json, or is_service_definition) specifically to codegen schema discovery.

Fixed in 54bc773 — explicit single-file paths now return immediately without running the directory-walk suffix filter.

damaz91 · 2026-10-01T13:43:06Z

+    let base = if s.contains('.') && !s.ends_with(".json") {
+        capability_short_name(s)
+    } else {
+        s.to_string()
+    };
+
+    base.split(['_', '-', ' ', '/'])
+        .filter(|part| !part.is_empty())
+        .map(|part| {
+            let mixed = part.chars().any(|c| c.is_ascii_uppercase())
+                && part.chars().any(|c| c.is_ascii_lowercase());
+            let mut chars = part.chars();
+            let Some(first) = chars.next() else {
+                return String::new();
+            };
+            let rest = if mixed {
+                chars.as_str().to_string()
+            } else {
+                chars.as_str().to_ascii_lowercase()
+            };
+            format!("{}{rest}", first.to_ascii_uppercase())
+        })
+        .collect()


Two edge cases in to_pascal_case:

Non-reverse-domain dotted keys: Using s.contains('.') && !s.ends_with(".json") drops the prefix on 2-segment dotted $defs keys such as "checkout.complete_request" (which appears in PR feat(codegen): add directional schema slicing and $ref alignment #82's tests), collapsing it to "CompleteRequest" instead of "CheckoutCompleteRequest". Using is_reverse_domain_name(s) here and including '.' in the base.split(...) delimiters would preserve non-RDN dotted identifiers while still extracting capability_short_name for RDN capability names.

Idempotency on consecutive uppercase letters: mixed checks any(uppercase) && any(lowercase). If an identifier has a single-letter segment followed by another segment (e.g., "a_b" -> "AB") or an all-uppercase acronym, calling to_pascal_case a second time sees mixed == false and lowercases the tail (to_pascal_case("AB") -> "Ab").

Fixed in 54bc773 — to_pascal_case now checks is_reverse_domain_name(s) before stripping a domain prefix, splits on '.', and treats delimiter-free strings starting with an uppercase ASCII letter as already PascalCase.

damaz91 · 2026-10-01T13:43:06Z

+pub fn is_generic_def_name(name: &str) -> bool {
+    const GENERIC: &str = "base|Base|entity|Entity|platform_schema|PlatformSchema|business_schema|BusinessSchema|response_schema|ResponseSchema|request|Request|response|Response|error|Error|config|Config|endpoint|Endpoint|quantity|Quantity";
+    !name.is_empty() && GENERIC.split('|').any(|g| g == name)
+}
+
+/// Qualify a hoisted `$defs` entry name with `parent_pascal` when `def_key` is generic
+/// or represents a container operation definition.
+pub fn qualify_def_name(parent_pascal: &str, def_key: &str) -> String {
+    let def_pascal = to_pascal_case(def_key);
+    if is_generic_def_name(def_key) || is_generic_def_name(&def_pascal) {
+        if parent_pascal.is_empty() || def_pascal.starts_with(parent_pascal) {
+            def_pascal
+        } else {
+            format!("{parent_pascal}{def_pascal}")
+        }
+    } else if (parent_pascal.ends_with("Search") || parent_pascal.ends_with("Lookup"))
+        && (def_key.ends_with("_request") || def_key.ends_with("_response"))
+    {
+        qualify_container_op_name(parent_pascal, def_key)
+    } else {
+        def_pascal
+    }
+}


Idempotency hazard with PascalCase entries in is_generic_def_name:

In the downstream pipeline (PRs #82/#83), rewrite_refs_to_defs is invoked more than once on the same schema AST (e.g., during upfront hoisting in hoist_defs and again inside slice_directional_schemas -> normalize_def_schema).

If a schema (say Checkout, parent_name = Some("Checkout")) contains a cross-file $ref to a standalone file whose stem is in this generic list (e.g., "types/config.json", "types/error.json", or "types/quantity.json"):

Pass 1: ref_to_def_name("types/quantity.json", Some("Checkout")) resolves via file_stem_to_pascal to "Quantity", rewriting the $ref to "#/$defs/Quantity".

Pass 2: ref_to_def_name("#/$defs/Quantity", Some("Checkout")) now enters the strip_prefix("#/$defs/") branch, and because is_generic_def_name("Quantity") is true for PascalCase "Quantity", it re-qualifies the already-rewritten cross-file reference into "#/$defs/CheckoutQuantity".

Similarly, if a hoisted $defs entry with directional annotations (e.g., in Profile) is rewritten in hoist_defs with parent_name = Some("Profile") and then passed to slice_directional_schemas(&prepared, hoisted_name) where normalize_def_schema passes parent_name = Some(hoisted_name), any ref that resolved to an unqualified PascalCase generic name would be re-qualified with hoisted_name on the second pass.

Suggestion: Either restrict is_generic_def_name to raw (lowercase snake_case) $defs keys ("base|entity|platform_schema|...") and remove || is_generic_def_name(&def_pascal) in qualify_def_name, or short-circuit ref_to_def_name when a #/$defs/<Target> key already starts with an uppercase ASCII letter.

Fixed in 54bc773 — ref_to_def_name now short-circuits #//<Key> when <Key> starts with an uppercase ASCII letter so already-rewritten cross-file references remain idempotent across multiple passes.

damaz91 · 2026-10-01T13:43:06Z

+pub fn ref_to_def_name(ref_str: &str, parent_name: Option<&str>) -> String {
+    if ref_str == "#" {
+        return parent_name.map_or_else(|| "Self".to_string(), to_pascal_case);
+    }


Nit: Consider checking if ref_str == "#" || ref_str == "#/" here so a root JSON Pointer "#/" doesn't fall through to line 117 (file_stem_to_pascal("")) and produce an empty "#/$defs/".

Fixed in 54bc773 — ref_to_def_name now treats both "#" and "#/" as root self-references.

damaz91 · 2026-10-01T13:43:06Z

+fn inspect_allof_props(obj: &Map<String, Value>, props: &mut BTreeSet<String>) -> (bool, bool) {
+    let mut has_props = false;
+    if let Some(Value::Object(map)) = obj.get("properties") {
+        props.extend(map.keys().cloned());
+        has_props = true;
+    }
+    let mut has_ref = obj.get("$ref").is_some_and(Value::is_string);
+    if let Some(Value::Array(all_of)) = obj.get("allOf") {
+        for branch in all_of.iter().filter_map(Value::as_object) {
+            let (bp, br) = inspect_allof_props(branch, props);
+            has_props |= bp;
+            has_ref |= br;
+        }
+    }
+    (has_props, has_ref)
+}
+
+fn prune_dangling_required(obj: &mut Map<String, Value>) {
+    if !obj.get("required").is_some_and(Value::is_array) {
+        return;
+    }
+    let mut declared = BTreeSet::new();
+    let (has_props, has_ref) = inspect_allof_props(obj, &mut declared);
+    let Some(Value::Array(reqs)) = obj.get_mut("required") else {
+        return;
+    };
+    if has_props && !has_ref {
+        reqs.retain(|v| v.as_str().is_some_and(|k| declared.contains(k)));
+    }
+    if reqs.is_empty() {
+        obj.remove("required");
+    }
+}


Bug (prune_dangling_required strips valid required fields inside nested anyOf / oneOf / then branches, affecting service.json):

strip_ucp_keywords calls prune_dangling_required(obj) inside for_each_schema_object_mut, which visits every subschema object in isolation. However, inspect_allof_props(obj) only inspects obj and its descendant allOf branches — it has no visibility into sibling $refs or properties declared on an enclosing object.

In the UCP schema corpus (source/schemas/service.json), $defs.platform_schema (ServicePlatformSchema) has:

"allOf": [ { "$ref": "#/$defs/base" }, { "required": ["spec"] }, { "anyOf": [ { "properties": { "transport": { "const": "rest" } }, "required": ["schema"] }, { "properties": { "transport": { "const": "mcp" } }, "required": ["schema"] }, { "properties": { "transport": { "const": "a2a" } } }, { "properties": { "transport": { "const": "embedded" } }, "required": ["schema"] } ] } ]

When for_each_schema_object_mut visits each anyOf branch object:

has_props is true (declared = {"transport"})

has_ref is false (because { "$ref": "#/$defs/base" } lives on the parent allOf, not inside the anyOf branch)

prune_dangling_required sees "schema" ∉ {"transport"} and deletes "required": ["schema"] from the rest, mcp, and embedded branches of ServicePlatformSchema (and similarly deletes "required": ["endpoint"] from the rest, mcp, and a2a branches of ServiceBusinessSchema in service.json).

The same issue occurs in any if/then/else or oneOf/anyOf branch that refines one property via "properties" while requiring another property declared on the parent schema.

Suggestion: Track (inherited_props, inherited_has_ref) top-down across same-instance combinators (allOf, anyOf, oneOf, if, then, else, not, dependentSchemas) while resetting to (empty, false) when stepping into nested instance schemas (properties/*, patternProperties/*, additionalProperties, items, prefixItems, $defs/*, definitions/*). Then only prune when local_has_props && !effective_has_ref, retaining keys in effective_props.

Fixed in 54bc773 — prune_dangling_required now recurses into nested object schemas while propagating inherited properties across same-instance combinators (allOf, anyOf, oneOf, if/then/else).

Implement Task 4 (PR 1d) of the ucp-schema codegen pipeline: - Wire Commands::GenerateTypes in src/bin/ucp-schema.rs with --profile, --capability, --extension, --schema-dir (alias --schema-local-base), --schema-remote-base, --output, and --pretty. - Implement CliExitCode for CodegenError and generalize write_json_output to <T: serde::Serialize + ?Sized>. - Add CLI integration tests in tests/generate_types_test.rs covering stdout pretty/compact output, file output, repeatable/comma-delimited flags, and error exit codes. Stacked PR Roadmap: - PR 1a (#81): Identifier qualification, UCP keyword stripping, $ref rewriting - PR 1b (#82): Directional schema slicing and $ref alignment - PR 1c (#83): Capability/extension reachability, composition, and generate_types() library API - PR 1d (this PR): CLI generate-types subcommand - PR 2: Self-contained .well-known/ucp --profile scoping - PR 3: Inline conditional variant hoisting and ordered anyOf union lowering - PR 4: OpenAPI 3.1 REST binding assembly (generate-openapi)

…ref, and prune vacuous anyOf stubs

Implement Task 4 (PR 1d) of the ucp-schema codegen pipeline: - Wire Commands::GenerateTypes in src/bin/ucp-schema.rs with --profile, --capability, --extension, --schema-dir (alias --schema-local-base), --schema-remote-base, --output, and --pretty. - Implement CliExitCode for CodegenError and generalize write_json_output to <T: serde::Serialize + ?Sized>. - Add CLI integration tests in tests/generate_types_test.rs covering stdout pretty/compact output, file output, repeatable/comma-delimited flags, and error exit codes. Stacked PR Roadmap: - PR 1a (#81): Identifier qualification, UCP keyword stripping, $ref rewriting - PR 1b (#82): Directional schema slicing and $ref alignment - PR 1c (#83): Capability/extension reachability, composition, and generate_types() library API - PR 1d (this PR): CLI generate-types subcommand - PR 2: Self-contained .well-known/ucp --profile scoping - PR 3: Inline conditional variant hoisting and ordered anyOf union lowering - PR 4: OpenAPI 3.1 REST binding assembly (generate-openapi)

…n in qualify_def_name

…irectory walks

gsmith85 requested review from amithanda, damaz91, igrigorik and jingyli September 30, 2026 16:10

damaz91 added the status:needs-triage Signal that the PR is ready for human triage label Sep 30, 2026

This was referenced Sep 30, 2026

feat(codegen): add directional schema slicing and $ref alignment #82

Open

feat(codegen): add capability reachability, composition, and generate_types() API #83

Open

damaz91 added status:under-review and removed status:needs-triage Signal that the PR is ready for human triage labels Oct 1, 2026

damaz91 reviewed Oct 1, 2026

View reviewed changes

gsmith85 mentioned this pull request Oct 1, 2026

feat(cli): add generate-types subcommand #84

Open

fix(codegen): strip rule-prose title fields from inline allOf subschemas

3549467

fix(codegen): preserve nested anyOf required, strip sibling type on $…

c75b1eb

…ref, and prune vacuous anyOf stubs

fix(codegen): generalize container op _request/_response qualificatio…

f1622aa

…n in qualify_def_name

gsmith85 mentioned this pull request Oct 7, 2026

feat: migrate model generation to ucp-schema generate-types Universal-Commerce-Protocol/js-sdk#87

Open

gsmith85 added 2 commits October 7, 2026 22:03

fix(linter): retain *.openapi.json and *.openrpc.json files in lint d…

17f4e13

…irectory walks

fix(codegen): address PR #81 review feedback in loader and normalizer

54bc773

Navigation Menu

Repository navigation

feat(codegen): add identifier naming, UCP keyword stripping, and $defs ref rewriting - #81

feat(codegen): add identifier naming, UCP keyword stripping, and $defs ref rewriting#81
gsmith85 wants to merge 6 commits into
mainfrom
feat/codegen-normalizer

gsmith85 commented Sep 30, 2026 •

edited

Loading

Uh oh!

damaz91 left a comment

Uh oh!

damaz91 Oct 1, 2026

Uh oh!

gsmith85 Oct 8, 2026

Uh oh!

damaz91 Oct 1, 2026

Uh oh!

gsmith85 Oct 8, 2026

Uh oh!

damaz91 Oct 1, 2026

Uh oh!

gsmith85 Oct 8, 2026

Uh oh!

damaz91 Oct 1, 2026

Uh oh!

gsmith85 Oct 8, 2026

Uh oh!

damaz91 Oct 1, 2026

Uh oh!

gsmith85 Oct 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

gsmith85 commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Stacked PR Series Roadmap

Test Plan

Uh oh!

damaz91 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gsmith85 commented Sep 30, 2026 •

edited

Loading