Benchmark methodology & setup

[Miloslav]

Description

Hi @PHPantom-dev team,

Congrats on the project! It's always great to see new tooling emerging in the PHP ecosystem.

We noticed the benchmark table in your README.md (21K PHP files, 1.5M LOC). Could you share which codebase or script was used for that 21K files benchmark, and how the test was executed?

Use case

We’d love to run these performance tests internally against PHP Tools to analyze potential bottlenecks and compare results under the exact same conditions and environment.

Thanks!
Best regards,
DEVSENSE Team

Proposed solution

No response

Alternatives considered

No response

Code example

[AJenbo]

Hi @Miloslav and the DEVSENSE team,

Thanks for reaching out, and thank you for the kind words! It’s great to see active interest and engagement across the PHP tooling ecosystem.

Here is a breakdown of our current benchmark setup, along with a proposal to build a transparent, reproducible testing standard.

The codebase referenced in the README is a private commercial Laravel application, so I am unfortunately unable to share that specific codebase publicly.

However, since the PhpStorm team has also reached out with similar questions, I would love for us to settle on a standard suite of open-source projects. Having an agreed-upon public benchmark would enable us to run fair, verifiable tests under identical conditions.

Benchmark Environment & Hardware

• OS: Ubuntu 26.04 LTS
• Hardware: 32-thread CPU
• Metric: CPU seconds (total CPU time consumed), rather than wall-clock time.

Why CPU time?
To eliminate the "works fast on my machine" problem, it also gives a good view of total energy usage. However, it does introduce specific trade-offs:

• Threading: Tools that utilize multi-threading poorly can appear artificially favorable in CPU time, even though a larger fraction of their execution time is felt directly as latency by the user.
• Overhead: On a 32-thread CPU, thread management overhead is more pronounced for tools that attempt to take advantage of a high core count.

Execution & Readiness Criteria

• PhpStorm: Caches are cleared, and the IDE is relaunched from the menu.
• LSPs: Caches are manually cleared, and the server process is launched cold via Zed.
• Measuring "Readiness": Time is tracked until total system resource consumption settles back to baseline idle and the system responds to a hover on code that utilizes the framework.

Known Nuances & Caveats:

• Background Indexing vs. Usability: Waiting for the system to settle can obscure when a server becomes functionally responsive. For instance, PHPantom becomes usable/interactive after ~1s, but continues background indexing across the workspace for ~4s.
• Full Diagnostics: On the current main branch, PHPantom runs full-project diagnostics in the background by default upon launch, which extends total CPU time to ~40s.

Ways to improve

It would be great to collaborate on selecting target OSS repositories and writing a reproducible benchmark script we can all run. If you have specific projects or methodology suggestions in mind, feel free to share them here.

I would prefer including a large open-source Laravel project in the suite, as framework-specific support is where we focus heavily and where LSPs spend significant time processing the codebase.

Best regards

[DarkGhostHunter]

In terms of OSS repositories, there are a lot of candidates, but I think there should be a selection of four:

1. Simple "Hello world" using a micro-framework (great to find problems)
2. A library (Wordpress is my candidate; great to most test code in source rather than in vendor)
3. A Symfony based project
4. A Laravel base project

I believe these four should suffice all scenarios, especially the last two in terms of framework usage.

[AJenbo]

Since there is no real way to "finish" this issue I'll convert it to a discussion.