Give the real-audio quality test a corpus

[thomas-vilte]

Description

TestEncoderQualityRealCorpus has been in the repository since the quality assurance system was implemented, but it has never been run: it is skipped unless OPUS_QUALITY_CORPUS points to a directory of clips, and there was no way to obtain one. The baseline against which it is compared was also never confirmed, so even a developer with their own clips would not get any regression checks; only a number was displayed once.

This script adds the missing piece. .github/scripts/fetch-quality-corpus.sh creates the corpus from a manifest of 12 live recordings from the Internet Archive's Live Music Archive, one artist per recording, verifying a SHA-1 for each source file. Only the URLs and checksums are in the repository, never the audio, which prevents licensing issues from being affected

by testdata/; This is what EBU SQAM ruled out when the control system was written.

The confirmed baseline was measured precisely with that corpus.

Clip Selection

Material that has already been processed by a perceptive codec is useless as a quality reference: re-encoding it aligned to the 20 ms grid it already contains causes quantization noise to overlap and the score to increase by approximately 1 dB.

Each clip was verified in two ways, as each verification method has a blind spot.

Frame sweeping—encoding at various sample offsets and looking for a periodic peak—
does not detect material with a very strong low-pass filter because there is no fine structure left to resonate with the grid.

Attempting to adopt clips from the 2011 HydrogenAudio listening test confirmed this:
the sweep ranked the most damaged half of that set as the cleanest, and the spectra showed an abrupt wall at 9 kHz. Thus, the second check is spectral and doesn't care about the block size used by the codec: a codec zeroes out everything above its cutoff frequency and leaves a plateau at the diming level, while the lossless material continues to descend to Nyquist.

The Live Music Archive was chosen because its elements document their provenance, so the provenance can be filtered before downloading and confirmed afterward.

Playability

The masters are 44.1 kHz, so the script sets soxr:precision=28 instead of using the default value from ffmpeg; a different resampler would change the baseline for reasons unrelated to the encoder. Everything else in the chain is already fixed: the source files using SHA-1, opus_compare using the SHA-1 published in the RFC, and the encoder is deterministic. Therefore, the committed values ​​should be reproducible everywhere, and the 15% safety margin absorbs any variation that might occur in a different soxr compilation.

At 96 kbps, the twelve clips exhibit a weighted error between 0.28 and 1.15, with a maximum of one defective window out of 2999 in any one of them.

Not integrated into CI

The script is deliberately kept as a local tool. Downloading the corpus is approximately 240 MB, which is a considerable expense per run for a check whose objective is to detect perceptual regressions during encoder work, and CI's quality assurance task already covers synthetic levels. The execution would look like this:

./.github/scripts/fetch-quality-corpus.sh /tmp/corpus
OPUS_QUALITY_CORPUS=/tmp/corpus OPUS_RFC6716_REFERENCE=... \
go test -tags conformance -run TestEncoderQualityRealCorpus .

Reference Problem

Part of the encoder quality work; not a separate problem.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.16%. Comparing base (c1f0a53) to head (b56e52e).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #212   +/-   ##
=======================================
  Coverage   93.16%   93.16%           
=======================================
  Files          58       58           
  Lines       10481    10481           
=======================================
  Hits         9765     9765           
  Misses        505      505           
  Partials      211      211           

Flag	Coverage Δ
go	93.16% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.