Add OTLP Metrics ingest support by gareth-ellis · Pull Request #2127 · elastic/rally

gareth-ellis · 2026-05-22T13:59:36Z

This commit adds support for indexing to the _otlp/v1/metrics endpoint of elasticsearch.

There's quite a bit more than just a new runner here, as follows:

To index to _otlp, you need an otel dataset. Currently I have created these via metricsgenreceiver.
Once you have a json otel dataset, you have two choices:
- Use the json doc as is, prior to a race starting that required an otlp dataset, then a .pb file will be created - this is the protobuf version of your json doc.
- Alternatively you can store this file along with your corpus, and the .pb file will be collected instead. This will save time, especially if running the benchmark multiple times.

gareth-ellis · 2026-05-29T14:36:20Z

This is split into 6 PRs now:
The recommended review/merge order applies:

#2134 and #2135 — fully independent, can merge in any order, in parallel.
#2136 and #2137 — both need #2135, can review in parallel once #2135 is in.
#2138 — needs #2135 + #2136 + #2137.
#2139 — needs #2138.

inqueue · 2026-09-16T17:48:59Z

+        * ``body``: Raw binary protobuf payload (bytes).
+
+        Optional parameters:
+        * ``endpoint``: OTLP endpoint path. Defaults to ``/_otlp/v1/metrics``.


Could a signal-type attribute be considered with possible values of metrics, logs, or traces instead of endpoint?

Copilot

🟡 Changes recommended

Retry parameters are dropped, prepared variants can resolve incorrectly, and corpus validation can accept corrupt or mismatched files.

12 open findings

Validate offset indexes against the protobuf corpus · New
Reject truncated records during corpus scanning · New
Reject negative retry values · New
Keep prepared OTLP format variants in the same directory · New
Filter corpora to the selected OTLP document set · New
Forward OTLP retry options from track parameters · New
Bound OTLP batches by cumulative bytes · New
Correct documented prepared filenames to include .json · New
Correct the documented batch count to 360 · New
Document installation of the OTLP conversion extra · New
Exclude retry waits from measured service time · New
Restore the missing test method boundary · New

What changed in this PR

Adds end-to-end OTLP metrics benchmarking through Elasticsearch’s native OTLP endpoint.

Changes:

Adds OTLP corpus conversion, preparation, partitioning, and gzip support.
Introduces the otlp-ingest operation with retries and metrics.
Adds dependencies, documentation, and comprehensive tests.

File	Description
`uv.lock`	Locks OTLP protobuf dependencies.
`pyproject.toml`	Defines OTLP and development dependencies.
`create-notice.sh`	Adds dependency license notices.
`esrally/utils/io.py`	Converts and reads protobuf corpus files.
`esrally/track/track.py`	Registers the OTLP format and operation.
`esrally/track/params.py`	Adds OTLP corpus partitioning and parameters.
`esrally/track/loader.py`	Adds format-aware corpus preparation.
`esrally/driver/runner.py`	Implements OTLP HTTP ingestion and retries.
`tests/utils/io_test.py`	Tests conversion and record indexing.
`tests/track/params_test.py`	Tests OTLP parameter sourcing.
`tests/track/loader_test.py`	Tests corpus preparation and parsing.
`tests/driver/runner_test.py`	Tests OTLP request and retry behavior.
`docs/track.rst`	Documents the new format and operation.
`docs/otlp_ingest.rst`	Adds the OTLP usage guide.
`docs/index.rst`	Includes the guide in documentation navigation.
`docs/advanced.rst`	Updates preparation architecture documentation.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

gbanasiak · 2026-10-08T16:49:31Z

+        # like bulk: trust an existing offset file and only verify when we have to build it
+        if os.path.exists(pb_file.pb_path + ".offset"):
+            return True


This is intentional. The same behavior is present in case of bulk corpus. The assumption is if someone uploaded offset file into the corpus store they want to reduce the corpus preparation time aggressively. We can change this assumption for both OTLP and bulk in unison but not in this PR.

Copilot

🟡 Changes recommended

Offline preparation and prepared-variant path resolution can prevent valid local OTLP corpora from running.

6 open findings

11 resolved since last review

🧠 Review effort: Balanced

+                    suffixes = DOCUMENT_SET_FORMATS[document_set.source_format].prepared_file_suffixes()
+                    resolved = first_existing_with_any_suffix(data_root, document_set.document_file, suffixes)


+                pb_archive_path = pb_path + archive_ext
+                try:
+                    preparator.downloader.download(document_set.base_url, pb_archive_path)
+                except exceptions.DataError:


+            if len(data_root) == 1:
+                preparator.prepare_document_set(document_set, data_root[0], fmt)
+            # attempt to prepare everything in the current directory and fallback to the corpus directory
+            elif not preparator.prepare_bundled_document_set(document_set, data_root[0], fmt):
+                preparator.prepare_document_set(document_set, data_root[1], fmt)


+When a challenge runs ``otlp-ingest`` with multiple clients (``"clients": N``), Rally splits the corpus across clients so each client reads a distinct, non-overlapping slice of the records. Partitioning uses the ``.pb.offset`` index for O(1) seek to each client's starting record.
+
+Each client's slice size is ``floor(total_records / N)``; the final client gets any remainder. This guarantees each record is sent exactly once per pass across all clients.


… include the complete source path' Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

gareth-ellis added 10 commits May 22, 2026 15:49

Bump dependencies, update notice

2edb3d4

Add support for otetl corpora, and otlp runner.

ffa107e

Update tests

2f63d5a

Dont crash on binary response error

31fe25d

Allow downloading compressed pb files

40b08c1

Improve efficiency of generating .pb corpus from json

1b33d4b

Test fix for errors in runner

0aa1ae2

Fix issue when local json doesnt exist

03924ca

Fix corpus reader

92235f9

Support gzipped otlp requests

bb9100b

inqueue requested changes Sep 16, 2026

View reviewed changes

gbanasiak and others added 10 commits October 6, 2026 22:26

Merge branch 'master' into pr/gareth-ellis/2127

ce895f2

Remove unused io methods

d651d2e

Verify record number

5d3d159

Increase loader extensibility

3988d77

Align OTLP and bulk param sources, drop deprecated size()

c232240

Add otlp docs

353643b

Update documentation

3cde51e

Remove endpoint param in runner

9050920

Simplify record counting

f13dc68

Reuse bounds calculation in params

783811e

gbanasiak mentioned this pull request Oct 8, 2026

Add otlp docs #2148

Closed

Drop is_otlp property

19b8707

gbanasiak requested a balanced review from Copilot October 8, 2026 16:39

Copilot started reviewing on behalf of gbanasiak October 8, 2026 16:39 View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

gbanasiak added 2 commits October 8, 2026 21:22

Address docs feedback and other docs improvements

25658d6

Handle truncated PB file

1384e64

gbanasiak changed the title ~~Add OTLP support to rally~~ Add OTLP Metrics ingest support Oct 8, 2026

gbanasiak added 7 commits October 8, 2026 22:35

Prepare all in the same directory

6bb6008

Fix and validate params

64e732b

Restore swollen test

fa101a7

Reuse corpus filtering in bulk and OTLP param sources

2525c32

Limit batches by bytes not by line count

6cc7005

Define .offset once

1e31ca3

Add logs/traces readiness

b51f5ee

gbanasiak requested a balanced review from Copilot October 9, 2026 21:17

Copilot started reviewing on behalf of gbanasiak October 9, 2026 21:18 View session

Copilot AI reviewed Oct 9, 2026

View reviewed changes

Potential fix for pull request finding 'Correct prepared filenames to…

70f09e6

… include the complete source path' Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Navigation Menu

Repository navigation

Add OTLP Metrics ingest support - #2127

Add OTLP Metrics ingest support#2127
gareth-ellis wants to merge 31 commits into
elastic:masterfrom
gareth-ellis:otlp

gareth-ellis commented May 22, 2026 •

edited by gbanasiak

Loading

Uh oh!

gareth-ellis commented May 29, 2026

Uh oh!

inqueue Sep 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

gbanasiak Oct 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		suffixes = DOCUMENT_SET_FORMATS[document_set.source_format].prepared_file_suffixes()
		resolved = first_existing_with_any_suffix(data_root, document_set.document_file, suffixes)

		When a challenge runs ``otlp-ingest`` with multiple clients (``"clients": N``), Rally splits the corpus across clients so each client reads a distinct, non-overlapping slice of the records. Partitioning uses the ``.pb.offset`` index for O(1) seek to each client's starting record.

		Each client's slice size is ``floor(total_records / N)``; the final client gets any remainder. This guarantees each record is sent exactly once per pass across all clients.

Conversation

gareth-ellis commented May 22, 2026 • edited by gbanasiak Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gareth-ellis commented May 29, 2026

Uh oh!

inqueue Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

gbanasiak Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gareth-ellis commented May 22, 2026 •

edited by gbanasiak

Loading

gbanasiak Oct 8, 2026 •

edited

Loading