Skip to content

Conversation

@pinylin
Copy link

@pinylin pinylin commented Dec 25, 2025

Summary

This PR adds support for multiple ClickHouse endpoints in the ClickHouse sink, enabling high availability and load balancing across multiple ClickHouse instances. The changes include:

  • Adding a new endpoints configuration option that accepts a list of ClickHouse server URLs
  • Deprecating the existing endpoint option in favor of endpoints
  • Adding health check configuration through the new distribution option
  • Implementing load balancing and automatic health checking across endpoints
  • Updating all integration tests to use the new endpoints option
  • Updating documentation to reflect the changes

Vector configuration

[sinks.clickhouse]
type = "clickhouse"
inputs = ["source_name"]
endpoints = ["http://clickhouse-1:8123", "http://clickhouse-2:8123", "http://clickhouse-3:8123"]
table = "logs"
database = "default"
compression = "none"

[sinks.clickhouse.distribution]
retry_initial_backoff_secs = 1
retry_max_duration_secs = 3600

[sinks.clickhouse.batch]
max_events = 1000
timeout_secs = 1

For backward compatibility, the deprecated endpoint option still works:

[sinks.clickhouse]
type = "clickhouse"
inputs = ["source_name"]
endpoint = "http://localhost:8123"  # deprecated but still functional
table = "logs"
database = "default"

How did you test this PR?

  • Unit Tests: Updated existing unit tests to use the new endpoints option
  • Integration Tests: Tested in a production-like ClickHouse cluster environment:
    • Cluster Setup: 3-shard, 2-replica ClickHouse cluster
    • Failover Testing: Stopped one replica nodes to verify:
      • Automatic failover to healthy endpoints
      • Health check logic properly detecting unhealthy nodes
      • Load balancing redistributing traffic to remaining healthy nodes
      • Recovery when nodes are brought back online
  • Compilation Tests: Verified that the code compiles without errors
  • Changelog Validation: Used the project's changelog validation script to ensure proper formatting
  • Backward Compatibility: Tested that the deprecated endpoint option still works with proper deprecation warnings

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • make fmt
      • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
      • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

@pinylin pinylin requested review from a team as code owners December 25, 2025 10:20
@github-actions github-actions bot added domain: sinks Anything related to the Vector's sinks domain: external docs Anything related to Vector's external, public documentation labels Dec 25, 2025
@github-actions
Copy link

github-actions bot commented Dec 25, 2025

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@pinylin
Copy link
Author

pinylin commented Dec 25, 2025

I have read the CLA Document and I hereby sign the CLA

Copy link
Contributor

@rtrieu rtrieu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a minor suggestion, but otherwise looks good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants