Internal telemetry metrics dispatcher

[andborja]

It is required to implement configurable components to process and export internal telemetry data.

The new component will collect metrics from the MetricsRegistryHandle as a snapshot, send it to internal processors in sequence and then export the processed data to the different exporters configured.

This component will run every "flush_interval".

The internal processors and internal exporters are configured from a set of implemented components.

An example of the configuration of the internal telemetry is:

settings:
  telemetry_config:
    metrics:
      flush_interval: 10s
      processors:
      - type: processor1
      - type: processor2
      exporters:
      - type: logging
      - type: prometheus
        config:
          host: "localhost"
          port: 8080
nodes:
...

The example presents the configuration of flush_interval for metrics, two processors in sequence and two different exporters with their own configuration.

[andborja]

This is related to #1236, as it might be the enabler of different exporters.

[jmacd]

Thanks @andborja. This is a good scope of work to bring our telemetry support forward.

Please use the JSON Schema defined for telemetry config across OTel, https://github.com/open-telemetry/opentelemetry-configuration, i.e., the same configuration struct used in the Collector under service::telemetry.

It's OK to leave this under settings::telemetry here, but we should start with that structure.

[jmacd]

I've discussed this with @andborja and OTel-Rust maintainers. It would be ideal if the OTel-Rust SDK supported the declarative configuration model developed by OpenTelemetry, but we think it should not be our responsibility to implement it.

The most important thing we agreed is that we will continue the "plan of record" and use the OTel-Rust SDK. We will bridge the multivariate metrics produced by the otap-dataflow telemetry library to the asynchronous observer model supported by the Metrics SDK.