test: [expected failures] function-library fixtures pending spec decisions by leongdl · Pull Request #165 · OpenJobDescription/openjd-specifications

leongdl · 2026-08-02T02:39:56Z

Important

Expected-failure / spec-question fixtures — do not expect these to pass. Parked in EXPR/proposed/ (not discovered by the runner). Each asserts a strawman expectation and must not move into jobs/ until the spec question beside it is resolved or the bug is fixed. Details in proposed/README.md.

Contents (3 fixtures, dual-implementation verified)

expr2.2.6--repr-py-newline-roundtrip — implementation bug in BOTH implementations: repr_py("a\nb") emits a raw newline inside the quotes (invalid Python literal; ast.literal_eval raises SyntaxError). §2.2.6 is explicit that repr_py follows Python's repr, which escapes. Move to jobs/ once fixed.
expr2.2.4--center-odd-padding — spec gap, NO divergence: both implementations put the odd pad space on the RIGHT (matching Python str.center). §2.2.4 is silent on the split. The spec should document the de facto behaviour; then this fixture (with corrected expectation) goes live.
expr2.2.4--isdigit-unicode — spec ambiguity, NO divergence: both implementations return false for isdigit("٣") (ASCII-only digits). §2.2.4 leaves "digit" undefined while §2.2.5 gives \d explicit Unicode semantics. Note the Python openjd CLI here deviates from native Python str.isdigit. The spec must define "digit"; the de facto agreement is ASCII-only.

Companion to the green-fixture PR from branch conformance-func-lib-gaps.

leongdl · 2026-08-15T17:48:36Z

Quorum verdict: NOT-GOOD (4/5 — three independent NOT-GOODs converged on the README) — spec: EL §2.2.4 center() is silent on the odd-split side, so 'spec question' is the right classification. But the README entry is wrong on the facts: (1) it claims both implementations return right-heavy ' hi ' with 'no divergence' — the 2026-08-12 sweep shows openjd-rs changed (upstream #305 + RFC 0005 coercion sync) and now emits CPython's ' hi '; only the Python CLI is right-heavy now (fails PYTHON only); (2) it misstates CPython — verified: 'hi'.center(7) == ' hi ' (extra space LEFT; left = pad//2 + (pad & width & 1) is not Python's formula); (3) it declares 'the fixture's left-heavy expectation was wrong' while the committed fixture still asserts ' hi ' — which now matches both CPython and openjd-rs. Action: rewrite the README entry to the sweep facts; the fixture's expectation can stand as the de facto pin, with the spec decision (document the CPython split) as the promotion gate.

leongdl · 2026-08-15T17:48:38Z

Quorum verdict: GOOD-WITH-NITS — spec: EL §2.2.4 leaves 'digit' undefined while §2.2.5 gives \d explicit Unicode semantics — correctly classified as a spec ambiguity, and the README's dual-run (both implementations ASCII-only, agreeing) matches the sweep. Nits: (1) the fixture header claims 'live implementation divergence' — there is none; both agree, the divergence is vs host-language str.isdigit — align the header with the README; (2) the strawman asserts Unicode-true, the OPPOSITE of the de facto agreement — if the spec ratifies ASCII (likely), the expectation flips; two reviewers suggest either flipping now or leaving this out of the inventory entirely since it prejudges an open spec decision rather than pinning a bug.

leongdl · 2026-08-15T17:48:40Z

Quorum verdict: GOOD (5/5, one stale comment) — spec: EL §2.2.6 (repr_py follows Python repr; the spec's own example escapes \n) — genuinely mandated, and the README's 'bug in BOTH implementations, dual-run verified' matches the sweep. The ast.literal_eval round-trip asserts semantics rather than byte-exact escaping — well built for promotion. Fix: the fixture's inline comment still says 'bug in openjd-rs' — update to both.

leongdl · 2026-08-15T17:48:43Z

Quorum review (5 independent agents). 3 fixtures + README. Net: 1 GOOD, 1 GOOD-WITH-NITS, 1 NOT-GOOD requiring a README rewrite before merge. The center-odd-padding README is internally contradictory and factually wrong about CPython (verified by execution: 'hi'.center(7) == ' hi ', extra space LEFT); per the 2026-08-12 sweep, openjd-rs now matches CPython and the PYTHON CLI is the outlier — the fixture's committed expectation actually matches the de facto answer, and the README is the defect. Cross-PR blocker: this branch and PR #164 both create EXPR/proposed/README.md (verified add/add conflict).

…sed/) Three parked fixtures with dual-implementation observations: center() odd-padding and ASCII-only isdigit() are spec gaps where BOTH implementations agree (no divergence - the spec should document the de facto behaviour), and repr_py raw-newline emission is a bug in BOTH implementations per the explicit spec text. Details in proposed/README.md. Companion to conformance-func-lib-gaps. Review: quorum-review fixes — corrected the center-odd-padding entry on the measured facts: CPython's str.center('hi',7) is ' hi ' (extra space LEFT, verified by execution), current openjd-rs matches CPython (re-verified against an upstream/main build this session), and the Python CLI is now the right-heavy outlier — the fixture's committed expectation is the de facto answer and the earlier README/header claims were wrong. isdigit reframed as spec ambiguity with NO divergence between the openjd implementations (both ASCII-only; the divergence is vs host CPython). repr_py attribution corrected to BOTH implementations. Kind-level proposed/ placement with a family-named README (README-func-lib.md) to avoid add/add conflicts with co-located expected-failures PRs. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>

leongdl · 2026-08-15T19:29:07Z

Quorum-review fixes applied and pushed (rebased onto mainline 3069673):

expr2.2.4--center-odd-padding README/header rewritten on the measured facts (the NOT-GOOD): CPython's 'hi'.center(7) is ' hi ' — extra space LEFT (verified by execution, three independent runs this review); current openjd-rs matches CPython (re-verified against an upstream/main build: emits ' hi '); the Python openjd CLI is now the right-heavy outlier. The fixture's committed expectation was correct all along — the old README's "both implementations are right-heavy / the expectation was wrong" claims are retracted in the new text.
expr2.2.4--isdigit-unicode: header reframed as spec ambiguity with no divergence between the openjd implementations (both ASCII-only; the divergence is vs host CPython), and the strawman's flip-on-ASCII-ratification path is documented.
expr2.2.6--repr-py-newline-roundtrip: stale "bug in openjd-rs" comment corrected to BOTH implementations.
Restructured to kind-level parking (EXPR/jobs/proposed/) with a family-named README (README-func-lib.md) — removes the add/add merge conflict with test: [expected failures] EXPR fixtures needing implementation fixes #164.

mwiebe · 2026-09-09T22:34:58Z

+          - -c
+          - |
+            print(r'ISDIGIT_ARABIC:{{ isdigit("٣") }}')
+            print(r'ISDIGIT_DEVANAGARI:{{ isdigit("३") }}')


This should be working now with OpenJobDescription/openjd-rs#341 merged

mwiebe · 2026-09-09T22:37:54Z

+# The expected: block below asserts the CPython/openjd-rs behaviour, which
+# has become the de facto answer. Do NOT move this fixture into jobs/ until
+# the spec picks a side ("follow CPython" would make this promotable and the
+# Python CLI's split a plain bug).


In #177 we went with matching Python whenever there was ambiguity about it, I think that's a good approach to follow here too.

Both were parked as expected failures. openjd-rs#374 landed the fix, so they pass and become the regression tests the proposed/README promised. Measured against a binary built from openjd-rs main at 5b04959, and against a pre-fix binary as the control: fixture pre-fix merged EXPR/jobs/expr2.2.6--repr-py-newline-roundtrip FAIL PASS WRAP_ACTIONS/.../wrap-repr-py-escapes-newline FAIL PASS Both pre-fix failures are "SyntaxError: unterminated string literal" with the expected output missing, which is the defect they were written for: repr_py quoted its output without escaping, so a value carrying a newline emitted it raw and CPython would not parse the result. Discovered by the runner's directory scan from their new locations, not just single-file mode, which is what leaving proposed/ was for. No regression in either category: WRAP_ACTIONS 73 passed 0 failed, EXPR 352 passed 0 failed. The EXPR fixture comes from the func-lib expected-failures branch (OpenJobDescription#165), which should drop its copy so the two do not collide on merge. Headers rewritten, since both described the parked state and claimed the implementations fail. Now spec, test, expectation in three paragraphs each, naming the paired fixture, within the comment-line budget. proposed/README.md and the proposed/ directory are removed: the fixture it described was its only entry. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>

openjd-rs#374 landed the repr_py escaping fix, so expr2.2.6--repr-py-newline-roundtrip passes and no longer belongs in proposed/. It is promoted to EXPR/jobs/ on OpenJobDescription#162 alongside its WRAP_ACTIONS twin, where the two land together and the control run against a pre-fix binary is recorded. Removing it here so the two copies do not collide when both merge. Its README section goes with it. The other two fixtures on this branch are untouched and still parked: both are spec decisions, not implementation bugs. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>

Both were parked as expected failures. openjd-rs#374 landed the fix, so they pass and become the regression tests the proposed/README promised. Measured against a binary built from openjd-rs main at 5b04959, and against a pre-fix binary as the control: fixture pre-fix merged EXPR/jobs/expr2.2.6--repr-py-newline-roundtrip FAIL PASS WRAP_ACTIONS/.../wrap-repr-py-escapes-newline FAIL PASS Both pre-fix failures are "SyntaxError: unterminated string literal" with the expected output missing, which is the defect they were written for: repr_py quoted its output without escaping, so a value carrying a newline emitted it raw and CPython would not parse the result. Discovered by the runner's directory scan from their new locations, not just single-file mode, which is what leaving proposed/ was for. No regression in either category: WRAP_ACTIONS 73 passed 0 failed, EXPR 352 passed 0 failed. The EXPR fixture comes from the func-lib expected-failures branch (OpenJobDescription#165), which should drop its copy so the two do not collide on merge. Headers rewritten, since both described the parked state and claimed the implementations fail. Now spec, test, expectation in three paragraphs each, naming the paired fixture, within the comment-line budget. proposed/README.md and the proposed/ directory are removed: the fixture it described was its only entry. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>

* test: Add expected-failure WRAP_ACTIONS fixtures (parked in proposed/) Spec-correct fixtures that FAIL against current reference implementations, parked in a proposed/ directory the conformance runner does not discover, so merging keeps the suite green. Each entry in proposed/README.md records the observed output, spec citation, and classification. Promote each fixture up one directory unchanged when its implementation fix lands. Companion to conformance-wrap-actions-gaps. Review: quorum-review fixes — kind-level proposed/ placement (jobs/proposed/); README corrected: the defect fails BOTH implementations (2026-08-12 sweep), the §5.2 ArgString conflict is now the stated promotion gate rather than a footnote (the fixture's own args are arguably spec-invalid until that is resolved), and the unit-level repr_py twin in the func-lib expected-failures PR is cross-referenced. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> * test: Promote the repr_py fixtures out of proposed/ Both were parked as expected failures. openjd-rs#374 landed the fix, so they pass and become the regression tests the proposed/README promised. Measured against a binary built from openjd-rs main at 5b04959, and against a pre-fix binary as the control: fixture pre-fix merged EXPR/jobs/expr2.2.6--repr-py-newline-roundtrip FAIL PASS WRAP_ACTIONS/.../wrap-repr-py-escapes-newline FAIL PASS Both pre-fix failures are "SyntaxError: unterminated string literal" with the expected output missing, which is the defect they were written for: repr_py quoted its output without escaping, so a value carrying a newline emitted it raw and CPython would not parse the result. Discovered by the runner's directory scan from their new locations, not just single-file mode, which is what leaving proposed/ was for. No regression in either category: WRAP_ACTIONS 73 passed 0 failed, EXPR 352 passed 0 failed. The EXPR fixture comes from the func-lib expected-failures branch (#165), which should drop its copy so the two do not collide on merge. Headers rewritten, since both described the parked state and claimed the implementations fail. Now spec, test, expectation in three paragraphs each, naming the paired fixture, within the comment-line budget. proposed/README.md and the proposed/ directory are removed: the fixture it described was its only entry. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> * test: Assemble the repr_py round-trip FAIL marker at run time windows-latest failed the Python lane on this fixture and ubuntu-latest passed the same installed CLI. Not an implementation difference: the fixture matched its own forbidden marker. The harness scans a job's whole output for forbidden substrings, and on Windows openjd-sessions-for-python echoes the full child command line at INFO (_subprocess.py logs list2cmdline(self._args)); the POSIX path logs only the temp .sh path. This fixture's script source contained the literal FAIL, and FAIL is its forbidden entry, so the echo matched even though every assertion passed. The run log discriminates the two cases. It reports "Found forbidden output: FAIL" with no "Missing expected output" line, so NEWLINE:PASS was printed and the escaping worked on Windows too. A genuine failure of this fixture looks different: the pre-fix control run reported the missing NEWLINE:PASS line alongside a SyntaxError. Split the marker across two adjacent literals, the convention this repo already uses for the same hazard (20c1a84, and the timeout fixture's 'SHOULD_NOT' + '_PRINT'). Python concatenates them at compile time, so the run-time value is unchanged at 'FAIL got=' while the source no longer carries the literal. Verified both halves: the value still compares equal, and the script body no longer contains FAIL. Verified the marker still catches a real failure: comparing against a wrong value makes the runner report 0 passed, 1 failed, and restoring the fixture byte-identically returns it to 1 passed, 0 failed. The Rust lane was never affected, because it prints only COMMAND_OUTPUT records while the command echo is tagged FILE_PATH|PROCESS_CONTROL. The WRAP_ACTIONS fixture is unaffected too: it declares no forbidden entries. Windows itself is unverified from here. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com> --------- Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>

This was referenced Aug 15, 2026

test: Add 20 function-library fixtures incl. repr_* round-trips #159

Open

test: repr_py control-character conformance tests #162

Merged

test: [expected failures] EXPR fixtures needing implementation fixes #164

Draft

leongdl commented Aug 15, 2026

View reviewed changes

leongdl force-pushed the conformance-func-lib-expected-failures branch from 4fd8d2b to 6fc5ab9 Compare August 15, 2026 19:26

mwiebe reviewed Sep 9, 2026

View reviewed changes

Navigation Menu

Uh oh!

test: [expected failures] function-library fixtures pending spec decisions - #165

test: [expected failures] function-library fixtures pending spec decisions#165
leongdl wants to merge 2 commits into
OpenJobDescription:mainlinefrom
leongdl:conformance-func-lib-expected-failures

leongdl commented Aug 2, 2026

Uh oh!

leongdl Aug 15, 2026

Uh oh!

leongdl Aug 15, 2026

Uh oh!

leongdl Aug 15, 2026

Uh oh!

leongdl commented Aug 15, 2026

Uh oh!

leongdl commented Aug 15, 2026

Uh oh!

mwiebe Sep 9, 2026

Uh oh!

mwiebe Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

leongdl commented Aug 2, 2026

Contents (3 fixtures, dual-implementation verified)

Uh oh!

leongdl Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

leongdl Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

leongdl Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

leongdl commented Aug 15, 2026

Uh oh!

leongdl commented Aug 15, 2026

Uh oh!

mwiebe Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

mwiebe Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants