feat: add initial opentelemetry tracing to big query HTTP requests

[ldetmer]

This feature adds the ability to enable open telemetry tracing on all HTTP requests.

This PR only contains the initial basic general/http attributes. Separate PRs will contain additional attributes.

Tested via sample test program and validated attributes show up in cloud trace:
https://screenshot.googleplex.com/AQJp4Nbb6oVbgAk

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the BigQuery client library by adding initial OpenTelemetry tracing capabilities to its HTTP requests. The changes enable the client to automatically generate and enrich OpenTelemetry spans for each HTTP call, providing deeper insights into the performance and behavior of interactions with the BigQuery service. This feature is conditionally applied, ensuring that tracing is only active when explicitly enabled.

Highlights

• OpenTelemetry Tracing Integration: Introduced initial OpenTelemetry tracing for BigQuery HTTP requests, allowing for detailed observability of network calls.
• HttpTracingRequestInitializer: Added a new HttpTracingRequestInitializer class that wraps existing HTTP request initializers to inject OpenTelemetry spans, capturing request and response details.
• Telemetry Attributes: Defined common OpenTelemetry attribute keys for GCP client information, HTTP request/response details, and error/exception handling within the BigQueryTelemetryTracer utility.
• Error and Exception Handling: Implemented logic within the tracing initializer to record error status codes, exception types, and messages in OpenTelemetry spans for failed HTTP requests.
• Comprehensive Testing: Included both unit and integration tests to ensure the correct functioning of the HTTP tracing, verifying attribute capture for successful, error, and exceptional scenarios.

Changelog

• google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java
  • Modified to conditionally wrap the HttpRequestInitializer with HttpTracingRequestInitializer if OpenTelemetry tracing is enabled and a tracer is provided.
• google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java
  • Added a new class to intercept HTTP requests and create OpenTelemetry spans.
  • Implemented logic to capture HTTP method, URL, host, port, and request/response body sizes.
  • Included mechanisms to record response status codes, error messages, and exceptions within the spans.
• google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/telemetry/BigQueryTelemetryTracer.java
  • Added a new utility class to define standard OpenTelemetry attribute keys for GCP client, error, and server information.
  • Provided a helper method for creating SpanBuilder instances with default BigQuery client attributes.
• google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/spi/v2/HttpTracingIntegrationTest.java
  • Added an integration test to validate HTTP tracing with a real HTTP server, confirming span creation and attribute population for successful requests.
• google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializerTest.java
  • Added unit tests for HttpTracingRequestInitializer to verify correct attribute setting for success and error responses.
  • Tested exception recording when the original unsuccessful response handler throws an IOException.
  • Confirmed that the delegate HttpRequestInitializer is called as expected.

Activity

• The pull request was opened by ldetmer.
• The author used a standard pull request template, outlining prerequisites such as opening an issue, passing tests, maintaining code coverage, and updating documentation.
• The PR aims to add initial OpenTelemetry tracing to BigQuery HTTP requests, as indicated by the title.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces OpenTelemetry tracing for BigQuery HTTP requests, enhancing observability. However, the current implementation has security concerns, as it leaks potentially sensitive information (PII, identifiers, and query details) into telemetry span attributes via full URLs and raw error messages. Furthermore, a critical bug allows spans to be ended multiple times, potentially corrupting telemetry data, and a separate issue in HttpBigQueryRpc.java leads to attribute loss. Addressing these security vulnerabilities and bugs, along with improving adherence to OpenTelemetry semantic conventions and completing TODO items, is crucial for accurate and secure telemetry.

[gemini-code-assist]

The OpenTelemetry attribute keys being defined here are based on an older version of the semantic conventions. The conventions have been stabilized and it's recommended to use the new key names to ensure compatibility with standard OpenTelemetry tooling.

For example:

• http.request.method should be http.method
• http.response.status_code should be http.status_code
• http.request.body.size should be http.request_content_length
• http.response.body.size should be http.response_content_length

Consider adding a dependency on io.opentelemetry:opentelemetry-semconv and using the constants from io.opentelemetry.semconv.trace.attributes.SemanticAttributes to stay aligned with the specification. If you prefer to avoid the extra dependency, please update the string keys manually.

  public static final AttributeKey<String> HTTP_REQUEST_METHOD =
          AttributeKey.stringKey("http.method");
  public static final AttributeKey<String> URL_FULL = AttributeKey.stringKey("url.full");
  public static final AttributeKey<String> URL_TEMPLATE = AttributeKey.stringKey("url.template");
  public static final AttributeKey<String> URL_DOMAIN = AttributeKey.stringKey("url.domain");
  public static final AttributeKey<Long> HTTP_RESPONSE_STATUS_CODE =
          AttributeKey.longKey("http.status_code");
  public static final AttributeKey<Long> HTTP_REQUEST_RESEND_COUNT =
          AttributeKey.longKey("http.request.resend_count");
  public static final AttributeKey<Long> HTTP_REQUEST_BODY_SIZE =
          AttributeKey.longKey("http.request_content_length");
  public static final AttributeKey<Long> HTTP_RESPONSE_BODY_SIZE =
          AttributeKey.longKey("http.response_content_length");

[ldetmer]

FYI I doubled, checked and it looks like what we're using is correct:

https://opentelemetry.io/docs/specs/semconv/http/http-spans/