Skip to content

benchmarks: Initial DirectTrajOpt benchmark suite (Ipopt vs MadNLP)#67

Open
jack-champagne wants to merge 8 commits intofeat/madnlp-integrationfrom
benchmarks/directtrajopt-initial
Open

benchmarks: Initial DirectTrajOpt benchmark suite (Ipopt vs MadNLP)#67
jack-champagne wants to merge 8 commits intofeat/madnlp-integrationfrom
benchmarks/directtrajopt-initial

Conversation

@jack-champagne
Copy link
Copy Markdown
Member

Summary

Initial benchmarking infrastructure for DirectTrajOpt.jl comparing Ipopt and MadNLP solvers on the feat/madnlp-integration branch.

Depends on: HarmoniqsBenchmarks.jl (new shared package — needs to be created in the org)

What's included

  • `benchmark/Project.toml` — Benchmark environment with HarmoniqsBenchmarks, BenchmarkTools, TestItems, MadNLP
  • `benchmark/benchmarks.jl` — Three `@testitem` suites:
    • Evaluator micro-benchmarks (bilinear N=51) — BenchmarkTools timing for each MOI eval function
    • Ipopt vs MadNLP macro-benchmark (bilinear N=51) — full-solve comparison
    • Memory scaling study — sweep N ∈ {25, 51, 101} × state_dim ∈ {4, 8, 16}
  • `docs/superpowers/specs/2026-04-15-benchmarking-design.md` — overall architecture
  • `docs/superpowers/specs/2026-04-15-altissimo-gpu-benchmarks-design.md` — 3-way GPU benchmark design
  • `docs/superpowers/plans/2026-04-15-benchmarking-infrastructure.md` — implementation plan

Preliminary Results

Ipopt vs MadNLP (bilinear N=51)

Solver Wall time Allocations
Ipopt 8.52s 3.4 GB
MadNLP 5.75s (33% faster) 1.9 GB (43% less)

Micro-benchmark hotspot

`eval_hessian_lagrangian` is the clear optimization target — 12.7ms median, 73K allocations, 68MB per call.

Test plan

  • Run `julia --project=benchmark -e 'using TestItemRunner; @run_package_tests(benchmark)'` — all 3 benchmarks produce JLD2 artifacts
  • Verify results load correctly via `load_results` / `load_micro_results`
  • Review preliminary Ipopt vs MadNLP timing numbers

🤖 Generated with Claude Code

jack-champagne and others added 6 commits April 15, 2026 01:00
Sets up benchmark/ with Project.toml, .gitignore, and three @testitem
benchmarks: evaluator micro-benchmarks, Ipopt vs MadNLP comparison, and
memory scaling sweep (N × state_dim grid).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Specs for:
- Overall HarmoniqsBenchmarks.jl architecture
- Altissimo GPU benchmarks (3-way: Ipopt CPU / MadNLP-GPU / Altissimo-GPU)
- Implementation plan for DirectTrajOpt + HarmoniqsBenchmarks tasks
- Add .github/workflows/benchmark.yml that runs on PRs touching src/ or benchmark/
- Uses Pkg.add(url=...) to install HarmoniqsBenchmarks (unregistered)
- Uploads JLD2 artifacts for 90 days
- Add benchmark/README.md with run instructions
- Remove empty BenchmarkUtils.jl leftover
- Ignore Manifest.toml (regenerated on each CI run)
Uses Julia 1.11+ [sources] section to resolve:
- DirectTrajOpt from local path (parent dir)
- HarmoniqsBenchmarks from public GitHub URL

CI workflow simplified to just Pkg.instantiate (no manual Pkg.add needed).
Remove docs/superpowers/ (specs and plans) from this PR to keep the
diff focused on benchmarks and MadNLP integration. Fix stale Pkg.add
instruction in benchmark README — deps resolve via [sources] now.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jack-champagne and others added 2 commits April 16, 2026 02:33
The benchmark @testItems require HarmoniqsBenchmarks which is only
available in the benchmark/ project environment, not the test extras.
Filter them out so `Pkg.test()` / julia-runtest CI doesn't pick them up.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant