Align KTO with DPO: Log entropy metric

[qgallouedec]

Part of the effort to align KTOTrainer (experimental) with DPOTrainer. #4786

DPOTrainer logs an entropy metric (mean per-token predictive entropy over completion tokens); KTOTrainer did not. Added it for parity, using the same token-weighted cross-rank averaging as DPO:

• Import entropy_from_logits from trl.trainer.utils.
• In _compute_loss, compute per-token entropy over the completion tokens (completion_mask), gather sums and token counts across ranks, and log the weighted mean as entropy.

Only added to _compute_loss, not _compute_loss_liger: the Liger path does not materialize logits, so entropy cannot be computed there (same as DPO).

No change to the loss or existing metrics.

---

Note

Low Risk
Observability-only metrics logging reusing the same helper and aggregation pattern as DPO; no loss or training logic changes.

Overview
Experimental KTOTrainer now logs mean per-token predictive entropy over completion tokens, matching DPOTrainer behavior for alignment work (#4786).

The change imports entropy_from_logits and, in _compute_loss, computes entropy from detached shift_logits, masks with completion_mask[:, 1:], and records a token-weighted mean after gather_for_metrics across ranks under the entropy metric key. KTO loss and existing metrics are unchanged.

Entropy is not added on the Liger path (_compute_loss_liger), because logits are not materialized there—same limitation as DPO.

^{Reviewed by Cursor Bugbot for commit 2fdb24f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.