Create a report by ARIA/HTML feature

[ccanash]

Top-Level Report Page Updates

On the reports page, after the introduction and before the L2 heading with text “Support Levels”, add a tab list with three tabs: “Test Plans”, “ARIA Features”, and "HTML Features."

The L2 heading that currently has content “Support Levels” will serve as a tab title and have content of either: “Test Plan Support Levels,” “ARIA Feature Support Levels,” or "HTML Feature Support Levels".

Table Summary by AT/browser combination

ARIA Summary Table example:

ARIA Feature	JAWS and Chrome	NVDA and Chrome	VoiceOver for macOS and Safari
aria-selected	X% of passing	X% of passing	X% of passing
button	X% of passing	X% of passing	X% of passing

HTML Summary Table example:

HTML Feature	JAWS and Chrome	NVDA and Chrome	VoiceOver for macOS and Safari
Button	X% of passing	X% of passing	X% of passing

Percentage and value definitions for AT/Browser Combo Support Level columns

Each percentage is a combination of must-have and should-have behaviors that should be combined into a single total assertion count (X% of passing = P/N*100):

1. N: Total number of “Must-Have” and "Should-Have" assertions that reference this attribute across all most-recent reports on this AT/browser combination where most-recent is:
   a. Report generated from most recent version of recommended test plan if available.
   b. If a recommended test plan report is not available, report generated from most recent version of candidate test plan if available.
   c. Data from draft test plans should not be included in attribute reports.
2. P: Number of N assertions with a passing verdict.
3. F: Number of N assertions with a failing verdict.
4. U: Number of N assertions that were untestable.

Additional Summary Table Features

1. The attributes link to specifications using data in support.json.
2. X% Support Level links to the raw data report described below.
3. Data can be downloaded in CSV format. TBD on how the values will be split across columns in the CSV.

Source Data

1. Each assertion, with some exceptions, has one or more associated ARIA attribute references, specified in “assertions.csv”.
2. The attribute being referenced is defined by the combination of the “type” and “value” columns in “references.csv”. Note that there are currently two types: aria and htmlAam. Reports by attribute should include both.
3. Each assertion with one or more references has verdicts from one or more commands in one or more tests in a test plan report.
4. A “support level” could be a percentage of passing verdicts from P1 and P2 (MUST and SHOULD) assertions, excluding P3 assertions. Note that the priority of a given assertion is context dependent, e.g., assertion A1 may have priority P1 for command C1 of test T1 and have priority P3 for command C2 of test T1.

ARIA/HTML Feature Detail Report

A new kind of report page that can be reached from links on the support level values in the top-level summary of ARIA/HTML feature support.

Page Title

Format:
AT_NAME and BROWSER_NAME Support for FEATURE_NAME | ARIA-AT Reports

Example:
JAWS and Chrome Support for aria-selected | ARIA-AT Reports

Page Content

1. Breadcrumbs
2. An L1 heading with content “AT_NAME and BROWSER_NAME Support for FEATURE_NAME” (FEATURE_NAME should link to spec derived from info in support.json)
3. An L2 heading "Summary of Results for AT_NAME and BROWSER_NAME" followed by a table of detailed breakdowns of # of passing, failing, and untestable assertions per priority level
4. A “Download CSV” button
5. A table of the raw data.

The assertions statistics summary should include the following columns and rows:

	Passing	Failing	Untestable
Must-Have Behaviors	6 of 12	3 of 12	3 of 12
Should-Have Behaviors	4 of 8	2 of 8	2 of 8
Must + Should	10 of 20	5 of 20	5 of 20
Percent of Behaviors	50%	25%	25%

The data table would contain the following columns:

1. Test Plan: Hyperlink with text of the name and version of the test plan
2. Test Title: Hyperlink that targets the heading for the test in the test plan report
3. Command: The command sequence as it appears in the test plan report
4. Assertion Priority: Must/Should
5. Assertion Phrase: The assertion phrase as it appears in the test plan report
6. Result: Passed|Failed|Untestable
7. Last Tested On: Date of the test as it appears in the test run history
8. Last Tested AT Version: Name/Version of AT as it appears in the run history in the test plan report
9. Last Tested Browser Version: Name/Version of browser as it appears in the run history in the test plan report
10. Number of severe side Effects: Count of severe side effects recorded for the command in the test plan report
11. Number of moderate side Effects: Count of moderate side effects recorded for the command in the test plan report

Detail report table features

1. Default sort order is ascending test plan name, test title, command
2. Given the CSV download, V1 of this report does not need sort, filter, or search functionality.

CSV File

The CSV file could have a name that reflects the page title:
AT_NAME+BROWSER_NAME-Support-for-FEATURE_NAME-ARIA-AT.csv

Example:
JAWS+Chrome-Support-for-aria-selected-ARIA-AT.csv

The first column of the report should be the ARIA/HTML Feature Name. This will disambiguate the content and enable people to easily combine multiple reports.

If a column is a hyperlink, the CSV file should have two columns – one for the link text and one for the Link URL.

[ChrisC]

@mcking65 I'm curious for more reasoning on the requirement for rendering 4 different values (if I'm understanding correctly) in the Support Level columns for the ARIA/HTML features:

X%
P/N passed
F/N failed
U/N untestable

Why not keep it consistent with the original test plan columns that only render "X% of passing verdicts" in a progress bar? I'm not sure it's all that helpful to display the additional values in this summary tables, and it also makes for inconsistent UI. Perhaps we can just include P/N passed, F/N failed, and U/N untestable in the report detail page?

[mcking65]

@ChrisC wrote:

I'm curious for more reasoning on the requirement for rendering 4 different values (if I'm understanding correctly) in the Support Level columns for the ARIA/HTML features:

X%
P/N passed
F/N failed
U/N untestable

Why not keep it consistent with the original test plan columns that only render "X% of passing verdicts" in a progress bar? I'm not sure it's all that helpful to display the additional values in this summary tables, and it also makes for inconsistent UI. Perhaps we can just include P/N passed, F/N failed, and U/N untestable in the report detail page?

I've been concerned about the transparency of what the percentage means. It is pretty difficult to derive it from the detailed page. This became more of a challenge when we added untestable. I was thinking that we might want to change the existing report summary to also show all 4 values in each cell.

I think we need to somehow make the calculations more transparent. If my original suggestion is too cluttered and not that helpful, then perhaps another approach would be to add an assertion statistics summary in a new section at the top of each detail page.

Maybe something like the following.

Summary of Results for NVDA and Chrome

	Passing	Failing	Untestable
Must-Have Behaviors	6 of 12	3 of 12	3 of 12
Should-Have Behaviors	4 of 8	2 of 8	2 of 8
Must + Should	10 of 20	5 of 20	5 of 20
Percent of Behaviors	50%	25%	25%

[ChrisC]

I've been concerned about the transparency of what the percentage means...

...If my original suggestion is too cluttered and not that helpful, then perhaps another approach would be to add an assertion statistics summary in a new section at the top of each detail page.

@mcking65 I like this suggestion for the assertion statistics summary at the top of each detail page! I'll add it to the requirements in the description!

[mcking65]

@ChrisC @howard-e @ccanash

(Revised after discovering data bug described in subsequent comment)

This is looking awesome in staging! Except for the one bug described in the next comment, It is definitely shippable. I have observed the following additional issues that should be either fixed before ship or have a separate issue raised so they can be addressed later.

On the raw data pages:

1. id="main" is missing from the main element so the skip to main link does not work.
2. The link in the test plan column goes to the report for that test plan rather than the test plan. I think that is OK, possibly better than the specified design. However, I suggest we change the title of the column to "Test Plan Report".
3. I suggest we change the name of the "Test Title" column to "Test"
4. The links in the test title column don't work for me. The right page loads, but the fragment is not scrolling to the specific test. The focus ends up on the first link.

I did not find any issues on the top-level report page or in the CSV file; they are fantastic.

As a result of these reports, I am observing some errors in test plan references. I'll get busy fixing those.

[mcking65]

Unfortunately, I've found a bug in the queries. The correct values are not being used for assertion priority. Frequently, the same assertion can have different priorities in different contexts, sometimes for different commands. The queries are not using the correct priority for each test/command/assertion combination.

These differences would be easier to understand and spot if we addressed issue #1588, which I just raised as I was thinking about how to improve the accuracy of ARIA/HTML feature mappings. I was thinking about how to prevent report accuracy problems caused by incorrect references like those I fixed with w3c/aria-at#1320.

I sniffed a problem when checking the accuracy of the aria-label report for NVDA. The summary stated 0/0 must-have assertions. It didn't make sense to me that there would be 0 must assertions for aria-label. Note that the aria-label report has:

Test Plan	Test Title	Command	Assertion Priority
Disclosure Navigation Menu Example (V24.10.10)	Navigate forwards to a collapsed disclosure button	Down Arrow (browse mode on)	Should

If you look at the test plan report, that assertion has priority "must". The default priority in assertions.csv for that assertion is 2. However, for the down arrow command in browse mode, the priority is 1.

Update after working on more test plan corrections

I've been through all the reports and haven't seen any "must" assertions; the summaries on the raw data pages all say 0/0 for musts. So, there is really something amiss.

[mcking65]

While working on correcting ARIA/HTML report accuracy with w3c/aria-at#1320 and w3c/aria-at#1321, I realized the app has a pretty significant problem with the transparency and understandability of mappings between assertions and ARIA/HTML features. To address this, I raised #1588.

[mcking65]

Related to, but not blocking for, this issue: #1589.

[mcking65]

It seems the top-level reports are using the values in the 'refId' column of the references.csv files to populate the 'ARIA Feature' and 'HTML Feature' columns. This is understandable but unfortunately causing a problem because the same feature could have different 'refId' values across the test plans. For example, the HTML-AAM mapping for a (represents a hyperlink) has a 'refId' of 'link' in some test plans and a 'refId' of htmlLink' in others.

Ideally, we would have specified a value for 'linkText' in the support.json file so that all links to the same feature have the same link text, and we would use that link text for the first column of the report ... but that's now hind sight.

I could create PRs to fix all the test plans that are inconsistent. However, that still leaves open the possibility of more inconsistency in the future. It would also be quite a bit of work because there are quite a few plans that reference the 'a (represents a hyperlink)' mapping in HTML-AAM.

Another possibility for automatically guaranteeing consistency would be to use the 'value' column from references.csv as the distinct name of the feature. There a few features for which that might look a bit cryptic, but that's not a bad tradeoff for now ... or maybe even for a long time.

I have found a couple of invalid values in the 'value' column, and I plan to create PRs to fix them, e.g., 'list' instead of 'ul'. I thought those values were getting checked automatically by the tests that run against test plan PRs during the build process ... apparently not, so that is a gap in our automated quality checking.

[mcking65]

I can't overstate the utility of the CSV downloads. They probably increased my productivity by 10 fold this weekend. I think they would also prove valuable on the top-level reports and on the test plan report pages.

Many, many thanks!!!!!!!!

[mcking65]

Two other observations about the reports in staging ...

• The test plans tab is out of order. it should be sorted alphabetically, but it appears to be n some random order.
• Several roles are missing from the ARIA features tab, e.g., alert, dialog, and menu. Possibly others. Just doing a mental spot-check as the number of roles seems on the low side when looking at the list of test plans.

[howard-e]

@mcking65 said:
I sniffed a problem when checking the accuracy of the aria-label report for NVDA. The summary stated 0/0 must-have assertions. It didn't make sense to me that there would be 0 must assertions for aria-label …

#1595 just got merged and should address the query issues. It was related to the priority management.

Several roles are missing from the ARIA features tab, e.g., alert, dialog, and menu. Possibly others. Just doing a mental spot-check as the number of roles seems on the low side when looking at the list of test plans.

Surprisingly the alert and modal-dialog test plans which are the only 2 to reference alert and dialog doesn’t include them in their respective assertions CSVs (but ‘menu’ should appear now):

• alert
• modal-dialog

Solving #1588 to display those mappings will make identifying those others clearer. w3c/aria-at#1325 now displays that in the PR previews and #1594 will do the same for aria-at-app:

• alert PR preview with feature mappings
• modal-dialog PR preview with feature mappings