Implement branch coverage by sferik · Pull Request #9751 · jruby/jruby

sferik · 2026-09-30T15:28:55Z

This adds the branches mode of the Coverage library. Results have the same shape, keys, numbering and counts as CRuby (its default, prism-based compiler):

Coverage.start(branches: true)
require "foo"
Coverage.result
# => { "/path/foo.rb" => { branches: { [:if, 0, 3, 4, 9, 7] => { [:then, 1, 4, 6, 4, 15] => 1, [:else, 2, 5, 4, 9, 7] => 2 }, ... } } }

Keys are [type, id, start_line, start_column, end_line, end_column] for each construct and [label, id, start_line, start_column, end_line, end_column] for each of its targets, with ids numbered consecutively across the file in the order CRuby's compiler meets them.

Addresses the branch half of #5147, following up on #9676, which added method coverage.

Every branching construct is covered: if, unless, elsif chains, ternaries, modifier forms, case/when, case/in, while, until (including begin/end while), and safe navigation calls and assignments. CRuby's corner cases are matched too:

empty arms and missing else clauses, and a clause's own span when its body is empty
elsif clauses reaching the shared end
loop bodies that never run (CRuby still compiles and reports them, so JRuby no longer drops them while branches are measured)
the nil path of every call in a &. chain (lazy &. chains no longer share a single nil exit)
parenthesized arms
conditionals on literal predicates, which CRuby folds away together with everything inside the arm that cannot run

BranchCoverage and BranchTarget hold one branching construct and the places it branches to. FileCoverage keeps them in declaration order and identifies them by type and span, so building the same source again (a reloaded file, a block turned into a method by define_method) finds the existing counters rather than declaring new ones.

The IR builder is where branches are marked, as suggested by @enebo in #5147 (comment), rather than recovered from the IR's own jumps:

each construct is declared while its IR is built, in the order CRuby declares it (an if after its predicate, a loop or case before, a safe navigation after its receiver), so nested constructs are numbered the same way
a CoverBranchInstr is emitted at the start of each arm and holds the counter directly. JIT-compiled code binds to it through an invokedynamic site (CoverageSite) keyed by file and target index, which becomes a no-op if the file is no longer being measured.
methods are built eagerly while branches are measured, so a file's result lists every branch, not only those in methods that happened to be called

Counting is a single lock-free atomic add (VarHandle#getAndAdd), as with method coverage. A result is built from a snapshot taken under the CoverageData lock, which reads and resets each count in one atomic step, so a branch reached while Coverage.result(clear: true) runs is reported once rather than dropped.

The parser now records a source span on every AST node. The generated parser gives each node the location of the production that produced it, widened by enclosing productions that hand the node along until an explicit span locks it (calls keep the span they were created with, so an attached block is not part of them), and empty productions get bison's zero-width location instead of a stale one. The grammar also records the extra positions branch keys need: where a predicate ends, where an else keyword starts, a when/in clause through its then, a modifier's statement as written, the parentheses around an arm, and whether a conditional is one CRuby folds. Both generated parsers were regenerated from RubyParser.y and skeleton.parser with jay.

Coverage.supported?(:branches) is now true and the "branch coverage is not supported" warning is gone.

Cc: @kares @headius

headius · 2026-10-01T10:11:31Z

Thanks! Will run this through the review mill.

sferik · 2026-10-01T16:04:20Z

@headius Thanks! Do you think this will also make it into the JRuby 10.1.3.0 release?

headius · 2026-10-01T21:32:43Z

@sferik Sure! JRuby 10.1.3.0 isn't scheduled right now. Perhaps a little motivation to release soon given some CVEs in BouncyCastle (mitigated by jruby-openssl 0.16.3) but we've got some momentum here to keep merging stuff.

headius

Looks like a good chunk. I've added some specific comments and change requests.

The complexity of the PR comes primarily from two places:

Adding parser and AST support for tracking more source information about branch arms.
Propagating branch information through all of the IR builder methods and adding compilation for branch coverage instructions when appropriate.

General idea is that AST builds up the data on the source ranges for branches and their arms, IRBuilder sets up the global state for tracking coverage against those branches while compile, and the branch coverage instructions (and jitted code) update that state.

BIG open question here for me is what this looks like with Prism. Eventually, we hope to adopt it as the default. Presumably what we get out of Prism already has this info, or it can be inferred.

There's room for refinement here but if we can address my comments this will be good to merge.

Coverage's branches mode now works and produces the same results as CRuby (its default, prism-based compiler): { file => { branches: { [:if, 0, 2, 2, 6, 5] => { [:then, 1, 3, 4, 3, 5] => 2, [:else, 2, 5, 4, 5, 5] => 1 }, ... } } } Every branching construct (if, unless, elsif chains, ternaries, modifiers, case/when, case/in, while, until, their modifier forms including begin/end while, and safe navigation calls and assignments) is reported with the same keys, locations, numbering and counts as CRuby, including its corner cases: empty arms and missing else clauses, a clause's own span when its body is empty, elsif clauses reaching the shared end, loop bodies that never run, nil paths of every call in a &. chain, parenthesized arms, and conditionals on literal predicates, which CRuby folds away together with everything inside the arm that cannot run. Design, following the same shape as method coverage: - BranchCoverage / BranchTarget hold one branching construct and the places it branches to, each target owning a lock-free atomic counter; FileCoverage keeps them in declaration order and identifies them by type and span, so building the same source again (a reloaded file, a block turned into a method) finds the existing counters rather than declaring new ones. A result is built from a copy taken under the CoverageData lock, which reads and clears each count in one atomic step, so a branch reached while Coverage.result(clear: true) runs is reported once, not dropped. - The IR builder declares constructs while it builds their IR, in the order CRuby's compiler meets them (an if after its predicate, a loop or case before, a safe navigation after its receiver), and emits a CoverBranchInstr at the start of each arm. The instruction holds the counter directly; JIT-compiled code binds to it through an invokedynamic site keyed by file and target index. Methods are built eagerly while branches are measured so a file's result lists every branch, and lazy &. chains no longer share a nil exit so each call's nil path counts. - The parser records a source span on every AST node: the generated parser now gives each node the location of the production that produced it (widened by enclosing productions that hand the node along, until an explicit span locks it; calls keep the span they were created with so an attached block is not part of them), and empty productions get bison's zero-width location instead of a stale one. The grammar records the extra positions branch keys need (where a predicate ends, where an else keyword starts, a when/in clause through its then, a modifier's statement as written, the parentheses around an arm) and whether a conditional is one CRuby folds. Coverage.supported?(:branches) is now true and the "branch coverage is not supported" warning is gone. Un-excludes the MRI branch coverage tests (including test_coverage_supported, test_coverage_ensure_if_return and test_tag_break_with_branch_coverage) and adds JRuby tests for keys and counts, result shape, suspend/resume/clear, exact counts under parallel calls, reloaded files and define_method blocks, and every execution mode. See jruby#5147

SimpleCov checks the branches it synthesizes for files that were never loaded against what Coverage reports, construct by construct, and on JRuby 18 of its 73 constructs disagreed with CRuby: - A literal predicate folds only where MRI's compiler folds it. Inside parentheses MRI keeps a node of its own, which folds only when it compiles to a single constant: (1; 2) and (a; [1]; 2) fold by their last statement when nothing before it compiles to anything, while (nil), ("x") and (-> {}) keep their branch. __ENCODING__ and a bare lambda fold. && and || follow which of the two labels can be jumped to, so a || 1 folds but nil && 1 does not. - Nothing in an arm that can never run is measured, which now includes the blocks, lambdas, classes and methods written there. - expr => pattern and expr in pattern report no branch, as from Ruby 3.4 on.

A construct's type and a target's label were strings, and both were looked up by a string built from them and their span. The type is now BranchCoverage.Type and the label BranchTarget.Label, each carrying the name Coverage reports it as, and declared constructs and targets are looked up by a record of the type or label and the span.

buildConditional took the arm that can never run as 0, 1 or -1. It is now IRBuilder.DeadArm: STATEMENTS, CONSEQUENT or NONE.

sferik · 2026-10-01T22:43:01Z

Thanks for the review! I've addressed the inline comments in 16a2e9c and c8a43c3.

On Prism. most of this PR doesn't depend on which parser is used. The counters (FileCoverage, BranchCoverage, BranchTarget), CoverBranchInstr, the JIT's CoverageSite, result building, and the branch handling in IRBuilder's conditional, loop, case and safe-navigation code all live in core. The hooks a Prism builder needs are already protected on IRBuilder: branchCoverageFile(), coverBranch() and the branches supplier of buildConditional.

That part should be simpler with Prism. The keys follow CRuby’s Prism compiler, so the positions come straight from Prism’s node locations (then_keyword_loc, ElseNode, ParenthesesNode and so on). None of the Node or grammar additions would be needed.

SimpleCov warned on every JRuby run without --debug that coverage may be inaccurate, citing jruby/jruby#1196 from the JRuby 1.7 days. JRuby's coverage no longer depends on full-trace mode. The whole suite on JRuby 10.0.7, run with and without --debug, reports every line of SimpleCov's own code identically: the same lines relevant, the same lines hit. With the branch and line coverage work in jruby/jruby#9751 and jruby/jruby#9682 the same holds for every line, branch, and method. --debug also slows JRuby down, so the warning only cost the people who followed it. Drop the warning, the advice in the troubleshooting guide, and JRUBY_OPTS=--debug from CI.

define_method(:new_name, some_method) adds a copy of some_method that keeps its name. MRI keys the coverage entry by the method's original_id, so for class A; def foo; end; end class B < A; define_method(:bar, A.instance_method(:foo)); end it reports [B, :foo, ...], not [B, :bar, ...], on every version from 3.3 to master (rb_coverage_method_data_of in thread.c). This reverts the keying by the added name from caf18c6, which assumed the opposite.

MRI gives a file new line and branch counters each time it is loaded (iseq_setup_coverage replaces the path's entry), while an eval that names an already covered file counts into that file's entry. JRuby merged every parse of a file into the same entry, so loading a file twice added the counts of both loads together. Only an eval now merges into an existing entry. A load clears the file's lines and branches; the branch targets stay registered, since code from the earlier load finds them by index. Method entries are kept, as MRI sums the calls of the methods defined at the same place.

headius

Not sure when we're going to get reviews from @enebo so I'll make the executive decision and go forward with this for now. We can do some testing to see how memory size is impacted. We do compile to IR but we do so lazily, leaving most AST nodes in memory until they will actually be executed.

enebo · 2026-10-09T20:47:36Z

@sferik @headius .... @headius and I talked a little bit about this and likely there will be a followup to move all those fields on Node/If/etc into record object(s) referenced by the node. AST size is substantial for almost all JRuby applications. A field reference to a record which is null when this feature is off will be much less that having all those fields present.

We could get more exotic and add an index value to every node as an 'int' (smaller than an object reference) and then store all this info into an arena alloc'd list of records. Since Prism is the future I think that is too much work on an AST which is going away. Also, I can see some memory management issues here without supporting sparse arrays (like deleting the AST but having to null out those nodes position info record(s) OR having too many nodes to fit in an int).

We do try and delete AST trees when we know we can but due to Ruby semantics we cannot delete as many as we would like so they hang around with the IR instructions.

headius · 2026-10-09T21:09:22Z

Also worth pointing out #9686 which will merge over our wrapper around the prism parser. Once that lands we can take a look at enhancements necessary to maintain coverage support with prism as well.

Branch and method coverage report where things are in the source, so the parser records a span on every node, and more on some: where the else of a conditional is, the parentheses around an expression, the statement of a modifier loop. All of it sat in fields of the nodes, which made every node 16 bytes larger, and a conditional or a loop up to 40, whether or not Coverage was in use. An AST is large in almost any application and much of it lives as long as the code does. As suggested in jruby#9751 (comment), a node now refers to a record holding its positions, and the parser only records them when Coverage has a use for them. Any other parse leaves that one reference null. - NodeSpan holds a node's span and its parentheses. Each production handing a node along widens its span, which replaces the record. Parsing CRuby 4.0's standard library with positions recorded takes about 5% more CPU time that way than with a mutable holder (0.200s against 0.191s, the median of seven interleaved rounds, and 0.185s with no positions recorded). - What a class of node keeps besides its span goes in a record of its own inside the NodeSpan, so it takes no field of the node: a SourcePosition for the else of a when or in clause, the end of a safe navigation and the arrow of a lambda, a SourceSpan for the body of a modifier loop, and a Branch for a conditional. - Branch coverage needs all of these. Method coverage only needs to know where definitions and blocks start and end, so that is all a parse records for it. - What decides the code that is compiled stays on the node, as flags: that an expression is parenthesized, that a conditional is a modifier, and what MRI folds of a literal condition. - A block and a method definition have their end line in a field of their own again, since every parse needs it. A parse decides what to record when it starts. If branch coverage starts before it ends, the file is measured with what was recorded, and so reports no branches. Instance sizes in bytes (compressed oops), before the coverage work, on the branch this one is based on, and now: Node 24 -> 40 -> 24 CallNode 40 -> 72 -> 48 FCallNode 32 -> 56 -> 40 IfNode 32 -> 80 -> 40 WhileNode 32 -> 72 -> 40 WhenNode 32 -> 64 -> 40 IterNode 40 -> 56 -> 40 LambdaNode 40 -> 56 -> 40 DefnNode 40 -> 64 -> 48 LocalVarNode 32 -> 48 -> 32 FixnumNode 32 -> 48 -> 32 SymbolNode 24 -> 48 -> 32 Most classes are still 8 bytes over what they were before coverage: the reference pushes them past an alignment boundary. With positions recorded, the branches and lines Coverage reports for the 727 files of CRuby 4.0's standard library are what they were, and Coverage.line_stub gives the same result for each of them whether or not positions are recorded. Tests check which nodes carry positions in each coverage mode, so that a parse without Coverage keeps none.

Branch and method coverage report where things are in the source, so the parser records a span on every node, and more on some: where the else of a conditional is, the parentheses around an expression, the statement of a modifier loop. All of it sat in fields of the nodes, which made every node 16 bytes larger, and a conditional or a loop up to 40, whether or not Coverage was in use. An AST is large in almost any application and much of it lives as long as the code does. As suggested in jruby#9751 (comment), a node now refers to a record holding its positions, and the parser only records them when Coverage has a use for them. Any other parse leaves that one reference null. - NodeSpan holds a node's span and its parentheses. Each production handing a node along widens its span, which replaces the record. Parsing CRuby 4.0's standard library with positions recorded takes about 5% more CPU time that way than with a mutable holder (0.200s against 0.191s, the median of seven interleaved rounds, and 0.185s with no positions recorded). - What a class of node keeps besides its span goes in a record of its own inside the NodeSpan, so it takes no field of the node: a SourcePosition for the else of a when or in clause, the end of a safe navigation and the arrow of a lambda, a SourceSpan for the body of a modifier loop, and a Branch for a conditional. - Branch coverage needs all of these. Method coverage only needs to know where definitions and blocks start and end, so that is all a parse records for it. - What decides the code that is compiled stays on the node, as flags: that an expression is parenthesized, that a conditional is a modifier, and what MRI folds of a literal condition. The first shares a byte with the flag for a newline that is only there for backtraces, rather than each being a boolean. That leaves the last byte of a node's own 24 free, and the JVM puts a flag of the node's class there, such as whether a call is lazy, instead of starting another 8 bytes for it. - A block and a method definition have their end line in a field of their own again, since every parse needs it. A parse decides what to record when it starts. If branch coverage starts before it ends, the file is measured with what was recorded, and so reports no branches. Instance sizes in bytes (compressed oops), before the coverage work, on the branch this one is based on, and now: Node 24 -> 40 -> 24 CallNode 40 -> 72 -> 40 FCallNode 32 -> 56 -> 40 IfNode 32 -> 80 -> 40 WhileNode 32 -> 72 -> 32 WhenNode 32 -> 64 -> 40 IterNode 40 -> 56 -> 40 LambdaNode 40 -> 56 -> 40 DefnNode 40 -> 64 -> 48 LocalVarNode 32 -> 48 -> 32 FixnumNode 32 -> 48 -> 32 SymbolNode 24 -> 48 -> 32 The reference takes 4 bytes, being a compressed pointer, and objects are aligned to 8. So a class either had the padding to take it or grew by 8: of the 128 classes of node, 64 are the size they were before coverage and 64 are 8 bytes over. With positions recorded, the branches and lines Coverage reports for the 727 files of CRuby 4.0's standard library are what they were, and Coverage.line_stub gives the same result for each of them whether or not positions are recorded. Tests check which nodes carry positions in each coverage mode, so that a parse without Coverage keeps none.

sferik mentioned this pull request Sep 30, 2026

Fix line coverage of statements inside string interpolation and heredocs #9682

Merged

sferik force-pushed the branch-coverage branch from 4d11dbb to 82332cc Compare October 1, 2026 21:51

headius requested changes Oct 1, 2026

View reviewed changes

sferik added 4 commits October 1, 2026 15:31

Name the dead arm of a folded conditional with an enum

c8a43c3

buildConditional took the arm that can never run as 0, 1 or -1. It is now IRBuilder.DeadArm: STATEMENTS, CONSEQUENT or NONE.

sferik force-pushed the branch-coverage branch from 82332cc to c8a43c3 Compare October 1, 2026 22:33

sferik added 2 commits October 2, 2026 05:25

headius approved these changes Oct 2, 2026

View reviewed changes

headius merged commit 0f2cc47 into jruby:master Oct 2, 2026
135 checks passed

sferik mentioned this pull request Oct 3, 2026

Match CRuby’s line coverage and branch locations #9770

Merged

headius added this to the JRuby 10.1.3.0 milestone Oct 8, 2026

sferik mentioned this pull request Oct 10, 2026

Keep coverage source positions off AST nodes when Coverage isn't in use #9795

Open

Navigation Menu

Repository navigation

Implement branch coverage - #9751

Implement branch coverage#9751
headius merged 6 commits into
jruby:masterfrom
sferik:branch-coverage

sferik commented Sep 30, 2026 •

edited

Loading

Uh oh!

headius commented Oct 1, 2026

Uh oh!

sferik commented Oct 1, 2026

Uh oh!

headius commented Oct 1, 2026

Uh oh!

headius left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sferik commented Oct 1, 2026 •

edited

Loading

Uh oh!

headius left a comment

Uh oh!

Uh oh!

enebo commented Oct 9, 2026

Uh oh!

headius commented Oct 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sferik commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

headius commented Oct 1, 2026

Uh oh!

sferik commented Oct 1, 2026

Uh oh!

headius commented Oct 1, 2026

Uh oh!

headius left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sferik commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

headius left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

enebo commented Oct 9, 2026

Uh oh!

headius commented Oct 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sferik commented Sep 30, 2026 •

edited

Loading

sferik commented Oct 1, 2026 •

edited

Loading