docs(opensearch): fix the ES-to-OpenSearch migration guide

[SuJinpei]

What

Fixes How_to_Migrate_from_Elasticsearch_to_OpenSearch.md. The guide was reviewed, then both migration paths were run end to end on a real cluster — ES 7.10.2 (3 nodes) → snapshot to S3 → OpenSearch 2.19.6 → reindex → upgrade to OpenSearch 3.3.1, plus ECK-deployed Elasticsearch 8.17.5 → reindex from remote. Everything below is either a defect that breaks a reader following the guide, or an explanation the run showed to be wrong.

Errors that break a reader following the guide verbatim

Problem	Effect
s3.client.default.endpoint: "<http://minio.example.com:9000>" in 3 samples	Markdown autolink brackets leaked into the YAML value — the S3 client is configured with an invalid host
repository-s3 URLs hardcoded to specific versions	The guide itself says the plugin version must match the node exactly; opensearch-plugin install refuses a mismatch and the node will not start
Init container image from an internal registry	Not pullable by customers
Keystore steps written as a single-pod action	The keystore is per-node; snapshots fail on the pods that were skipped
"indices": "*" while the note below advises excluding system indices	The command contradicts its own guidance
Mappings extracted with sed	Silently drops index settings — an index using a custom analyzer cannot be recreated at all
OpenSearch upgraded to 3.3.1, Dashboards to 3.3.0	Mismatched pair
Phase 1's cluster resource cannot start	Two separate omissions, each fatal — see below
version.created given as 7102099 for ES 7.10.2	The real value is 7100299; the reader never sees the documented number

Why Phase 1 could not start

• No spec.bootstrap.pluginsList. Every general.additionalConfig entry is rendered onto the bootstrap pod too, and s3.client.* are only valid settings once repository-s3 is installed, so the bootstrap pod dies with unknown setting [s3.client.default.region]. The other nodes are pinned to it via cluster.initial_master_nodes, so the cluster never forms. The prerequisites section already showed the bootstrap: block — Phase 1 did not.
• No admin password. OpenSearch 2.12+ exits with No custom admin password found. security.config.adminCredentialsSecret does not supply it — that is only the credential the Operator uses to reach the cluster — and the bootstrap pod has no env field, so it must go through additionalConfig.

Explanations corrected

• Reindex from Remote was presented as ES 8.x-only. It also works from ES 7.10 and can target OpenSearch 3.x directly, removing the entire two-hop (verified: 5/5 documents from ES 7.10.2 straight into 3.3.1). Added a Choosing a Method section describing the real decision — copying index files vs re-indexing documents — including that reindex carries no settings, mappings or aliases.
• The ES 8.x Lucene claim is unverified. The guide said ES 8.x snapshots are unreadable because of "a newer Lucene version with incompatible metadata protocols". Measured, ES 8.17.5 is on Lucene 9.12.0 and OpenSearch 2.19.6 on 9.12.3 — the same Lucene minor. An actual ES 8 snapshot restore was not tested, so rather than substitute another explanation the text now states the behaviour without asserting a cause.
• additionalConfig becomes environment variables on every pod, not entries in an opensearch.yml ConfigMap. Corrected everywhere it was described.

Added

• A Troubleshooting section for a node stuck in CrashLoopBackOff after an additionalConfig change: values there are not validated by the Operator, a bad key is only rejected at node boot, and the rollout halts at the first failed node while the rest keep serving the previous config.
• A warning that reindex.remote.whitelist was removed in OpenSearch 3.x in favour of reindex.remote.allowlist — a real failure observed on a 3.3.1 cluster:

  SettingsException[unknown setting [reindex.remote.whitelist] please check that any
  required plugins are installed, or check the breaking changes documentation for
  removed settings]
  

• What a restore does not carry over (index templates, ingest pipelines, ILM policies, users and roles, Kibana saved objects), replica counts on a smaller target cluster, and taking a snapshot before the major upgrade.
• Relative links to the sibling OpenSearch guides, following this directory's convention.

Structure

The ES 7.10 path is renumbered Phase 0 / Phase 1 / Phase 2. Previously the source-side snapshot steps sat in an unnumbered Procedure section between Prerequisites and Phase 1, and step numbering restarted three times.

Notes for reviewers

• English only. docs/zh is regenerated from the English source by the pipeline.
• The example versions (7.10.2, 2.19.3, 3.3.1) are unchanged, but are now explicitly labelled as examples to substitute, with the command to read the running version. Aligning literal version numbers across the whole OpenSearch KB set is a separate change.
• The admin-password route deserves a second opinion. Passing it through additionalConfig stores it in plain text in the cluster resource — the warning says so and points at securityConfigSecret for anything long-lived — but if there is a better supported route for OpenSearch 2.12+ on Operator 2.8.x, that belongs here instead.

[SuJinpei]

Reviewed 0f90dfb. Thanks for picking this up — four of the five edits land real gaps and I'd take them as-is. Three things I'd like resolved before merge, flagged inline:

1. The opensearch-k8s-operator#883 citation describes a failure mode that issue does not report. In #883 the env var was set correctly; the reporter passed a JSON array and OpenSearch parsed it into two mangled strings. The actionable warning that issue supports is "don't write a list-valued setting as an array" — which belongs next to the existing comma comment, not framed as the Operator dropping the entry.
2. _nodes/settings?filter_path=**.reindex* is the one command in this commit I could not confirm returns anything. If it comes back empty on a correctly-configured cluster, the doc manufactures a false alarm on exactly the step it is trying to de-risk.
3. "a different distribution produces a plugin the node refuses to load" — the plugin is a version-pinned zip and the node checks elasticsearch.version, which is identical between the -oss and default 7.10.2 images. The two real reasons (air-gap pullability, exact version match) are already stated elsewhere and both hold. This is the same class of defect as the Lucene rationale the first commit removed: an explanation that reads correctly but was never measured.

Plus two consistency points: the recommended security-config path is only in prose, never in the YAML people actually copy, and "short-lived migration cluster only" cannot be followed on the ES 7.10 path, where that cluster is upgraded in place and becomes production.

Good as-is, no changes needed: the admin-credentials → admin-credentials-secret rename now matches KB260100022 §1.5 exactly and is the only occurrence in the file; the "read the image off the running StatefulSet" command and the air-gap rationale are the right fix for the -oss example; scoping the env-var rendering to Operator 2.8.x is the correct instinct; the relative link target and the #troubleshooting anchor both resolve; build is green.

Not from this commit, still open: the operand examples are pinned at 2.19.3 / 3.3.1 while the packaged component ships 2.19.6 / 3.7.0, so the exact-match plugin URLs break on an air-gapped cluster installed from the package. The "replace every occurrence with the versions your own clusters run" warning covers it in principle.

[SuJinpei]

The mechanism here is not supported by anything we have measured.

The plugin is a version-pinned zip fetched from artifacts.elastic.co (or the internal mirror), and the node validates elasticsearch.version from plugin-descriptor.properties. elasticsearch-oss:7.10.2 and elasticsearch:7.10.2 are the same version, so the installed plugin directory is identical — the init container is only supplying the elasticsearch-plugin binary. When I ran this end to end I used the default distribution throughout and never saw a load refusal, and I can't find a distribution-specific check that would produce one.

The two reasons that do hold are already in hand:

• air-gap pullability — stated in the paragraph right below, and it's the strong one;
• version must match exactly — already a :::warning further up.

Suggest dropping the "produces a plugin the node refuses to load" clause and keeping the rest:

The init container must use the same Elasticsearch image your nodes already run — the plugin is installed into a volume that the node container then mounts, and the plugin refuses to load on a node running any other version. Read the image off the running workload and paste that exact value:

The -oss example was worth removing for a separate reason worth keeping in the commit message but not the doc: that distribution has no security plugin, so it contradicts the elastic user this guide authenticates as everywhere.

[SuJinpei]

Scoping this to Operator 2.8.x is right and I'd keep it.

The parenthetical is the part I'd cut. I checked, and I can't find a released Operator version that renders additionalConfig into a mounted opensearch.yml. The nearest thing is a maintainer proposal in opensearch-k8s-operator#883 ("IMO the best option forward is to rework how additionalConfig is provided … If we switch this to a configmap that gets mounted into the pods"). Presenting a proposal as current behaviour of some other version sends readers looking for a ConfigMap that has never existed — which is what the first commit deleted from Troubleshooting after verifying it on a live 2.8.0 cluster.

Suggest either naming the version that does this, or, if the hedge is worth keeping, marking it as upstream direction:

(Upstream is considering moving this to a mounted opensearch.yml ConfigMap — see opensearch-k8s-operator#883 — so re-check this on a newer Operator.)

[SuJinpei]

The prose below recommends the securityConfigSecret path, but this block — the thing readers actually copy — only implements the other one. There's no securityConfigSecret anywhere in the sample, so following the recommendation means reconstructing the whole block from KB260100022 by hand.

Suggest making both paths visible in the one artifact:

  security:
    config:
      # credentials the Operator itself uses to reach the cluster; the password must match
      # the admin password the cluster is initialized with (see the warning below)
      adminCredentialsSecret:
        name: admin-credentials-secret
      # Recommended: supply the full security configuration and drop the
      # OPENSEARCH_INITIAL_ADMIN_PASSWORD entry above - see KB260100022.
      # securityConfigSecret:
      #   name: securityconfig-secret

The secret names line up with KB260100022 §1.5 as written, so this is copy-paste compatible with that guide.

[SuJinpei]

"Short-lived migration cluster only" and "Delete the migration cluster … once the migration is finished" don't hold on this path.

The OpenSearch 2.x cluster deployed here is upgraded in place to 3.x in Phase 2 Step 2 and becomes the production cluster — deleting it is not one of the reader's options. So a reader who follows this section as written ends up with a plain-text admin password in a long-lived production resource, which is the outcome the caveat is trying to prevent. The "or rotate the password through the security configuration" half is the only applicable branch, and it's buried second.

Suggest replacing the closing line with something the ES 7.10 reader can act on:

On the ES 7.10 path this cluster is upgraded in place in Phase 2 and becomes your production cluster, so rotate the password through the security configuration (KB260100022) before you finish. Only delete it if you took the reindex-from-remote route into a separate target.

The verification command in this bullet is a good addition — keep it.

[SuJinpei]

This comment is correct and is where the #883 reference actually belongs.

That issue is a value-format bug, not a propagation bug: the reporter wrote reindex.remote.allowlist: '["host-a", "host-b"]' and OpenSearch parsed it into two mangled entries ("[\"host-a\"", "\"host-b\"]"). Writing a list-valued setting as a YAML/JSON array is the natural thing to try, and it is what actually bites people — so it deserves an explicit "don't":

      # Allow connections to the ES 8.x host. Host and port only - no http:// or https:// prefix.
      # Separate multiple hosts with commas. Do not write this as a YAML or JSON list:
      # additionalConfig values are passed through as plain strings, and an array is
      # parsed into mangled entries (opensearch-k8s-operator#883).
      reindex.remote.allowlist: "es8-cluster-host:9200"

The single-host form shown here is the one I confirmed working, as is a comma-separated pair.

[SuJinpei]

Two problems in this paragraph — the underlying idea (verify before you rely on it) is good and worth keeping.

1. #883 is misattributed. That issue does not report a setting failing to reach the nodes. The reporter's own follow-up is explicit: "the env var is set properly (and must exist as a string)" — the value arrived, and it was the JSON-array formatting that broke it. The maintainer response confirms the design issue is that additionalConfig values are untyped strings passed through env vars, not that entries get dropped. So "the entry did not reach the nodes (#883 tracks this)" describes a failure mode nobody has reported, and the suggested fallback — nodePools[].additionalConfig — goes through the same env-var rendering and would not have helped that reporter. I'd re-aim the paragraph at "verify it parsed the way you expect" and move the #883 citation up to the comma comment on the sample.

2. The command is unverified. _nodes/settings?filter_path=**.reindex* is not demonstrated anywhere I can find. The form shown working in #883 — twice, by two different people — is:

curl -k -u "admin:<password>" "https://localhost:9200/_cluster/settings?include_defaults=true&filter_path=**.reindex.remote.allowlist"

_nodes/settings should surface -E-injected settings and has the real advantage of being per-node, which matches the "every node must report" instruction — so if you can run it once against a cluster where the setting is applied and confirm it returns the value, keep it. If not, use the _cluster/settings form. An empty response from a command that never reports on a correctly-configured cluster turns this into a false alarm on the exact step it's meant to de-risk.

[SuJinpei]

This puts back the kubectl get cm | grep step the first commit removed. That removal wasn't a style call — I checked on a live Operator 2.8.0 cluster and there is no opensearch.yml ConfigMap at all; the only ConfigMap in the namespace containing an s3.client.* key was one I had hand-written myself. So on the version this product ships, step 3's fallback always finds nothing, and a reader debugging a CrashLooping node spends the step learning that.

Same ask as the other version-hedge: if there is a released Operator that does this, name it; otherwise drop these three lines, or mark the ConfigMap as upstream direction rather than an alternative to check today.