Derive the VBR floor from the coded bin count

[thomas-vilte]

Description

In VBR, the encoder was only using 28–38% of the requested bitrate. At 96 kbps it was averaging around 27 kbps.

The cause was the target floor in compute_vbr. libopus (celt_encoder.c) does:

bins = eBands[nbEBands-2]<<LM;
floor_depth = (opus_int32)SHR32(MULT16_16((C*bins<<BITRES),maxDepth), DB_SHIFT);
floor_depth = IMAX(floor_depth, target>>2);
target = IMIN(target, floor_depth);

pion had:

floorDepth := float32(channelCount*effectiveBytes*8) * maxDepth / 65536

There are two problems here. It was using effectiveBytes where libopus uses a bin count (eBands[19]<<LM = 480, not the byte budget), and it divided by 65536, which corresponds to the SHR32(..., DB_SHIFT) — an identity in the float build (arch.h:313).

Together, those made the floor several orders of magnitude too small, so IMIN(target, floor_depth) always reduced the target to target>>2: one quarter of the requested bitrate on every frame.

I also ported the transient boost, which was previously approximated as:

if transient {
    target += target >> 3
}

libopus scales it using tf_estimate and compensates for the average:

tf_calibration = QCONST16(0.044f,14);
target += (opus_int32)SHL32(MULT16_32_Q15(tf_estimate-tf_calibration, target),1);

SHL32 is also an identity in the float build, so this becomes:

target += (tf_estimate - 0.044) * target

Measurement

Average packet size over 200 music frames, compared against the requested bitrate:

bitrate	target	before	after	libopus
24000	60 B	22.71 (37.9%)	54.30 (90.5%)	53.50
48000	120 B	37.80 (31.5%)	111.97 (93.3%)	113.71
96000	240 B	68.34 (28.5%)	227.51 (94.8%)	244.03
128000	320 B	88.61 (27.7%)	304.47 (95.1%)	—

Constrained VBR was also under target at around 77–79%; it now reaches 94–95%.

The packet sizes still vary frame to frame as expected: there are 10–12 distinct sizes per bitrate, ranging from 51 to 60 bytes at 24 kbps.

What is still short

At 96 kbps, libopus averages 244 bytes, slightly above the nominal 240, while pion stops at 227.5.

That's a separate structural issue: pion currently gives CELT a budget equal to frameBytes, so unrestricted VBR can only undershoot the target, never exceed it. libopus separates the packet buffer limit from the nominal vbr_rate, allowing demanding frames to use more bits while the reservoir keeps the long-term average under control.

That also shows up in quality: VBR round-trip SNR against libopus changes by +0.19 dB at 24 kbps, −0.15 dB at 48 kbps, and −0.45 dB at 96 kbps. The direction follows the amount of bits being spent.

Separating those two budgets is a separate change, so I'm leaving it for another PR.

Tests

TestVBRTracksTargetBitrate catches the main issue: it averages 40 frames and requires the result to stay above 80% of the requested bitrate. With the old floor it was around 30%, so the test fails.

TestVBRProducesVaryingPacketSizes was still passing with the bug because the two synthetic tones happened to produce different packet sizes anyway. I rewrote it around a case where the variation actually depends on the target being correct: a silence frame versus tonal frames.

Reference issue
Part of #34.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.13%. Comparing base (a38c746) to head (afcab56).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #202      +/-   ##
==========================================
+ Coverage   93.10%   93.13%   +0.03%     
==========================================
  Files          58       58              
  Lines       10263    10260       -3     
==========================================
+ Hits         9555     9556       +1     
+ Misses        500      498       -2     
+ Partials      208      206       -2     

Flag	Coverage Δ
go	93.13% <100.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.