fix kj-test & wd-test test coverage reporting

[anonrig]

This pull-request ensures that Rust tests, wd_test and kj_test are generating proper test coverage (and with correct line coverages).

Current state of the test coverage on main branch can be accessed from https://app.codecov.io/gh/cloudflare/workerd

Improvements

Tests done to improve the speed of the test coverage runner:

• Tested different optimization levels for both deps and actual project.
• Enabled experimental_split_coverage_postprocessing which had the highest impact on reducing the coverage duration.
• Experiment with different v8-codegen-opt targets/options.
• Experiment with different optimization levels

Current Issues

Some ongoing issues with test coverage:

• It takes 2 hours to run on CI, hence it's better to run the coverage tests on main branch only.
• In order to remove, --experimental_outputs_writable flag, we need to land Fix COVERAGE_DIR permissions for split coverage post-processing bazelbuild/bazel#28311
• Rust tests are not properly generated after enabling experimental_split_coverage_postprocessing. In order to fix Rust test coverage, we need to land fix collect coverage script to work with recent changes bazelbuild/rules_rust#3812, and update our rules_rust.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.86%. Comparing base (d19e973) to head (8d45d41).

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #5881       +/-   ##
===========================================
- Coverage   68.10%   48.86%   -19.24%     
===========================================
  Files          29      397      +368     
  Lines        2593   105997   +103404     
  Branches       15    17965    +17950     
===========================================
+ Hits         1766    51797    +50031     
- Misses        827    45671    +44844     
- Partials        0     8529     +8529     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[fhanau]

Added a couple of suggestions – it's a lot but I hope they'll help with some of the issues you've encountered with coverage.

[fhanau]

nit: I'm not convinced this is needed – with remote_download_minimal you'd expect that coverage data is not downloaded, try using remote_download_toplevel (which is already the default so just ensuring that remote_download_minimal isn't being set anywhere should help). See https://blog.bazel.build/2023/10/06/bwob-in-bazel-7.html for more info on bazel caching modes.

[anonrig]

Under remote execution (which we don't use, but might still apply), it states that:

Bazel will fail to create coverage information if tests have been cached previously. To work around this, --nocache_test_results can be set specifically for coverage runs, although this of course incurs a heavy cost in terms of test times.

---

I'm not sure if this still holds or not.

[anonrig]

Ref: https://bazel.build/configure/coverage

[mikea]

+1 on extracting v8 (and pretty much every other external library) from coverage to get speed under control.

I am also quite surprised by the amount of tests excluded. They seem to be incredibly useful? We should try to have the build fast enough to include as much as possible.

[anonrig]

Rebased and force pushed.

[anonrig]

Temporarily removed main branch requirement of coverage CI.

[anonrig]

Marking it as draft for now. While reverting the wd_test changes the coverage started failing again. Unfortunately, custom rule doesn't play well with coverage and we need sh_test in order to generate coverage reports correctly. I'll try to find a common ground.