Backport of Fix for WAN federation policy update in secondaryDC to older version when stale reads from lagging follower in primaryDC into release/1.22.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #22954 to be assessed for backporting due to the inclusion of the label backport/1.22.

The below text is copied from the body of the original PR.

---

Ensure retry if policy batch read for updating into raft has stale reads.
Fixes the wan federation policy replication issue where raft committed older policy

Description

This issue happens when there is a mismatch in primaryDC state consistency between leader follower
and WAN federation setup where secondaryDC replicates the policies from primaryDC.

Testing & Reproduction steps

Tested with a script for 30 mins to see

• if any policy replication from primary was delayed (yes , delay happens and sometimes i had to retry once for fetching the policy from dc2 till it gets replicated)
• if policy replicated from primary is wrongly set (no, this fixes this) .

Links

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

PCI review checklist

• I have documented a clear reason for, and description of, the change I am making.

• If applicable, I've documented a plan to revert these changes if they require more than reverting the pull request.

• If applicable, I've documented the impact of any changes to security controls.

  Examples of changes to security controls include using new access control methods, adding or removing logging pipelines, etc.

---

Overview of commits

• 958227e

[github-team-consul-core-pr-approver]

Auto approved Consul Bot automated PR