Opensearch version and distribution check fails if request path is set

[ConaghFM]

Steps to replicate

Set the path parameter in the output plugin config (the block) to post requests to a specific index URL.
It is necessary to be able to do this for access control, but it causes the version/distribution check to make it's request to the index instead of the base URL, and does not get the version and distribution parameters it is requesting:

2022-07-17 10:24:50 +0000 [error]: #0 unexpected error error_class=OpenSearch::UnsupportedProductError error="The client is not supported for the provided version and distribution combination."
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-ruby-1.0.0/lib/opensearch.rb:78:in 'verify_version_and_distribution'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-ruby-1.0.0/lib/opensearch.rb:74:in 'verify_open_search'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-ruby-1.0.0/lib/opensearch.rb:47:in 'method_missing'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/opensearch-api-1.0.0/lib/opensearch/api/actions/info.rb:45:in 'info'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:518:in 'detect_os_major_version'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:510:in 'block in handle_last_seen_os_major_version'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/opensearch_index_template.rb:70:in 'retry_operate'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:507:in 'handle_last_seen_os_major_version'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-opensearch-1.0.3/lib/fluent/plugin/out_opensearch.rb:383:in 'configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/plugin.rb:187:in 'configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:132:in 'add_match'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in 'block in configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in 'each'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in 'configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:149:in 'configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in 'configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in 'run_configure'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:719:in 'block in run_worker'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in 'main_process'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in 'run_worker'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in '<top (required)>'
2022-07-17 10:24:50 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in 'require'
2022-07-17 10:24:50 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in 'require'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in '<top (required)>'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in 'load'
2022-07-17 10:24:50 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd:23:in '

'

Expected Behavior or What you need to ask

The plugin queries the base URL for version/distribution check, then changes to the specified path when posting write operations.

Using Fluentd and OpenSearch plugin versions

• OS version
  5.4.181-99.354.amzn2.x86_64
• Bare Metal or within Docker or Kubernetes or others?
  fluent/fluentd-kubernetes-daemonset:v1.14.6-debian-opensearch-1.0
• Fluentd v1.0 or later
  • paste result of fluentd --version or td-agent --version
    fluentd 1.14.6
• OpenSearch plugin version
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

*** LOCAL GEMS ***

addressable (2.8.0)
aws-eventstream (1.2.0)
aws-partitions (1.574.0)
aws-sdk-core (3.130.0)
aws-sigv4 (1.4.0)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (2.2.24, default: 2.1.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.10)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
domain_name (0.5.20190701)
etc (default: 1.1.0)
excon (0.92.2)
faraday (1.10.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.3)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
ffi (1.15.5)
ffi-compiler (1.0.1)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-concat (2.5.0)
fluent-plugin-detect-exceptions (0.0.14)
fluent-plugin-grok-parser (2.6.2)
fluent-plugin-json-in-json-2 (1.0.2)
fluent-plugin-kubernetes_metadata_filter (2.9.5)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.3)
fluent-plugin-parser-cri (0.1.1)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-systemd (1.0.5)
fluentd (1.14.6)
forwardable (default: 1.3.1)
gdbm (default: 2.1.0)
getoptlong (default: 0.1.0)
http (4.4.1)
http-accept (1.7.0)
http-cookie (1.0.4)
http-form_data (2.3.0)
http-parser (1.2.3)
http_parser.rb (0.8.0)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.6.1)
json (default: 2.3.0)
jsonpath (1.1.0)
kubeclient (4.9.3)
logger (default: 1.4.2)
lru_redux (1.1.0)
matrix (default: 0.2.0)
mime-types (3.4.1)
mime-types-data (3.2022.0105)
msgpack (1.5.1)
multi_json (1.15.0)
multipart-post (2.1.1)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
netrc (0.11.0)
observer (default: 0.1.0)
oj (3.11.0)
open3 (default: 0.1.0)
opensearch-api (1.0.0)
opensearch-ruby (1.0.0)
opensearch-transport (1.0.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
prime (default: 0.1.1)
prometheus-client (4.0.0)
pstore (default: 0.1.0)
psych (default: 3.1.0)
public_suffix (4.0.6)
racc (default: 1.4.16)
rake (13.0.6)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
recursive-open-struct (1.1.3)
reline (default: 0.1.5)
rest-client (2.1.0)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby2_keywords (0.0.5)
sdbm (default: 1.0.0)
serverengine (2.2.5)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
systemd-journal (1.4.2)
timeout (default: 0.1.0)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
unf (0.1.4)
unf_ext (0.0.8.1)
uri (default: 0.10.0)
webrick (1.7.0, default: 1.6.1)
yajl-ruby (1.4.2)
yaml (default: 0.1.0)
zlib (default: 1.1.0)

• OpenSearch version (optional)
  1.3.3

[Watson1978]

Thanks for the detailed report.

This turns out to be a limitation of how path currently works rather than a bug specific to the version check itself.

The path value is applied at the transport/client level, so it is prepended to every request the client makes — including the version/distribution check the plugin runs at startup (detect_os_major_version → client.info). Because of that, the info request is sent to <path>/ (your index) instead of the cluster root /, the response does not contain the version / distribution fields that opensearch-ruby expects, and verify_version_and_distribution raises UnsupportedProductError.

As a workaround, you can skip the startup version detection and tell the plugin which major version to assume:

<match your.tag>
  @type opensearch
  # ... your host / auth settings ...
  path /your/index/path
  verify_os_version_at_startup false
  default_opensearch_version 1
</match>

With verify_os_version_at_startup false the plugin no longer calls client.info, so the request that was failing against the path-prefixed URL is never made. Set default_opensearch_version to the major version of your cluster (1 for OpenSearch 1.x).

Making the startup check always query the base URL regardless of path would require the info request to bypass the path-prefixed client, which is not straightforward with the shared transport, so the workaround above is the recommended approach for now.