Upgrade to OpenSearch 3.0

[chibenwa]

Though OpenSearch 2.x will end Maintenance Support once 4.0 is released. There's thus no real emergency but we can anticipate and create a dedicated ticket,

CF #5442 (comment)

• Step 1: validate James compatibility
• Step 2: upgrade perf test environment and bench
• Step 3: document the upgrade
• Step 4: update Ansible playbooks
• Step 5: Rolling adoption across all platforms
  • LNG prod
  • CNB preprod
  • CNB prod
  • MU preprod
  • MU prod
  • TWP platforms (dev, preprod, prod, cozy)

[Arsnael]

apache#2762

[Arsnael]

update Ansible playbooks

I have been looking a while on our internal gitlab projects and honestly I can't find any ansible playbook at all for opensearch... I'm thinking it likely got deployed manually

@chibenwa want me to try to spend some time making one? or we just try to do a manual upgrade of our testing env?

EDIT: of course upgrading to 2.19.2 first. Does not seem to have any really breaking change server side for this upgrade

[Arsnael]

Opensearch current versions btw depending of platforms:

• testing sandbox: 2.4.1
• LNG prod: 2.11.1 (used by twp stg as well I think)
• cozy: 2.15.0
• cnb staging: 2.17.1
• cnb prod: dont have access to it
• mu preprod: 2.18.0
• mu prod: 2.15.0

[Arsnael]

Also adding it here: if we ever upgrade the opebnsearch servers to 3.x, from version 3 minimum jdk is 21. Need to keep it in mind, as I bet old machines running it probably dont have it

[chibenwa]

Can you confirm that with driver 3.x one can still use opensearch 2.x?

[Arsnael]

Can you confirm that with driver 3.x one can still use opensearch 2.x?

apache#2763

[Arsnael]

Upgrade opensearch sandbox: from 2.4.1 to 2.19.2 (for now)

Manual upgrade, on each node:

• backup opensearch install: cp -r /usr/share/opensearch /home/linagora/opensearch-2.4.1-bak
• download tarball opensearch 2.19.2:

wget https://artifacts.opensearch.org/releases/bundle/opensearch/2.19.2/opensearch-2.19.2-linux-x64.tar.gz
tar -xvf opensearch-2.19.2-linux-x64.tar.gz

• copy following conf of 2.4.1 into config folder of 2.19.2:
  • opensearch.yml
  • jvm.options and jvm.options.d/
  • log4j properties file
  • certificates
• following rolling upgrade opensearch doc on each node: https://docs.opensearch.org/docs/2.19/install-and-configure/upgrade-opensearch/rolling-upgrade/
  • step 6 : replace /usr/share/opensearch folder with the new 2.19.2 one and copied conf before restarting opensearch

It seems successful:

GET "/_cat/nodes?v&h=name,version,node.role,master" | column -t
name                  version  node.role  master
opensearch-master-02  2.19.2   dm         -
opensearch-master-01  2.19.2   dm         *
opensearch-master-03  2.19.2   dm         -

GET /_cluster/health?pretty
{
  "cluster_name" : "development-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 83,
  "active_shards" : 171,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Will start doing some tests first on this with 2.19.2

Then when I finish we can upgrade to v3 maybe @chibenwa ?

[chibenwa]

Ok

[Arsnael]

Been trying to do a bunch of runs but in general it ain't better...

v3 client works with 2.19.2 os server (no problem, regarding it is compatible)

Following runs I took out search overrides but the gatling scripts are the same as normal perf tests

jmap

v2 java client

v3 java client

Email/query is slower... mean is x2 slower and p99 is around 60% slower

p99: 258 vs 395
mean: 47 vs 116

imap

java v2 client

java v3 client

Way slower...

• searchDeleted: mean 1205 vs 1552, p99 8511 vs 13154
• searchUnseen: mean 821 vs 1071, p99 5649 vs 9366

I feel like still same issues as before whent rying to switch to the new java client with apache http transport instead of the rest client.

I've been trying to look how to keep the rest client way... but they stripped most of deps out of it from v3 and seems pushing toward the apache http5 client as well.

Had some issues with tests. Trying to find a way

[Arsnael]

i did tests as well with modifying the gatling scripts initially to push more on opensearch operations but even just on normal run there is quite a significant difference.

If you want them I can post though

[quantranhong1999]

Stupid question, but can the v2 client work with OpenSearch v3?

[Arsnael]

Stupid question, but can the v2 client work with OpenSearch v3?

Not a stupid question at all! Normally yes, I remember seeing a page on OS doc before mentioning it was compatible... cant figure it out today

I will try ultimately to upgrade the opensearch cluster to v3 in sandbox at some point and test this as well