fix(newmm): remove quadratic time in graph reset and BFS path copy

[kamthorn]

Summary

Removes two sources of superlinear time in newmm tokenization (one_cut) on the current main (2.0.0, unreleased):

1. graph.clear() costs O(capacity). one_cut pre-reserves graph to input_len / 10 entries and calls graph.clear() once per BFS resolution. HashMap::clear() walks the whole table, so each call costs O(capacity), and both the number of calls and the capacity grow with the text. The graph is now replaced with a fresh map (and the upfront reserve is dropped, since it is discarded at the first reset).
2. bfs_paths_graph copies the path at every step. path.clone() for each vertex pushed to the queue is quadratic on text that stays ambiguous over a long stretch (e.g. "กรรมกร" repeated). It now records one predecessor per vertex and rebuilds the path once, when the goal is reached. The queue holds CharacterIndex instead of (usize, Vec<usize>).

The visited-set fix from #101 is kept: each vertex is still enqueued at most once, and the returned path is the same first-reached (shortest) path.

Measurements

Rust core, words_th.txt (62k words), release build, seconds, median of runs. "Paragraph" is a fixed Thai/English paragraph repeated N times.

Input	main	this PR
paragraph x100 (60,800 chars)	0.0061	0.0049
paragraph x400 (243,200 chars)	0.0337	0.0189
paragraph x800 (486,400 chars)	0.122	0.040
paragraph x1000 (608,000 chars)	0.274	0.049
"กรรมกร" x2,000 (12,000 chars)	0.0081	0.0024
"กรรมกร" x8,000 (48,000 chars)	0.151	0.009
"กรรมกร" x32,000 (192,000 chars)	2.59	0.050

On the paragraph, main jumps from 0.12 s at 486k characters to 0.27 s at 608k. That is where the reserved table doubles (57,344 entries is the load limit of a 65,536-bucket table, and input_len / 10 goes from 48.6k to 60.8k), which is what points at clear(). For comparison, the 1.4.0 release takes 0.085 s on the 608k paragraph.

Verification

• Output compared byte for byte between main and this branch on: the adversarial texts above, the paragraph x1000, the whole dictionary concatenated in file order, 40,000 shuffled dictionary words, and 20,000 lines mixing three dictionary words with Latin/digits/newlines. 0 differences.

• cargo test (debug and --release), cargo fmt --check, cargo clippy --all-targets: pass.

• Two new regression tests in tests/test_tokenizer.rs, both timing-ratio based rather than absolute:

  • test_newmm_persistent_ambiguity_scaling: 4x longer "กรรมกร" text must take less than 10x as long (about 4x if linear, 16x if quadratic).
  • test_newmm_long_text_scaling: 10x longer ordinary text must take less than 25x as long.

  Both fail on main (0.30 s vs a 0.23 s limit, and 0.58 s vs a 0.34 s limit) and pass with this change.

Not covered: the Python and Node.js bindings, and safe_mode / parallel_mode timing, were not run. The existing tests cover their behavior through the Rust API only.

🤖 Generated with Claude Code