MDA-1305: Change Duplicate Detection Logic In TAF For FTX Records

[rnama]

Summary

We are making this change to prevent valid FTX records from being incorrectly dropped during the TAF build.

The current FTX duplicate-detection/final-claim matching logic does not include a date in its key, which can cause distinct claims in the same claim family to be treated as duplicates when multiple final claims exist. Other claims files avoid this issue by including the adjudication date in the key, but FTX does not contain an adjudication date.

To align FTX behavior with other claims files and preserve valid records, this change adds PMT_OR_RCPMT_DT (payment/receipt date) to the duplicate-detection key and final-claim matching logic, with null-safe handling.

Associated Jira Tickets

• MDA-1305
• MDA-1560

Scope

Changes Included

• Added PMT_OR_RCPMT_DT to the FTX deduplication key.
• Updated the join between deduplicated FTX records and claim-family records to include a null-safe payment date comparison using:

  coalesce(PMT_OR_RCPMT_DT, date('1960-01-01'))

• Normalized the claim-family payment date column by aliasing:

  PYMT_OR_RCPMT_DT AS PMT_OR_RCPMT_DT

  prior to joins.
• Normalized SUBMTG_STATE_CD using:

  upper(trim(SUBMTG_STATE_CD))

  to reduce case-related mismatches.

Implementation Reference

• FTX.py:57
• FTX.py:91
• FTX.py:115

Change Size Summary

Metric	Value
Files Changed	1
Insertions	8
Deletions	3

[Copilot]

Pull request overview

This PR updates the FTX segment’s duplicate-detection and final-claim matching logic to avoid dropping valid FTX records by including PMT_OR_RCPMT_DT (payment/receipt date) in the de-dup key and claim-family join, with null-safe comparisons.

Changes:

• Added PMT_OR_RCPMT_DT to the _nodups grouping key for FTX extraction.
• Updated the _nodups join-back and the claim-family join to include a null-safe payment-date comparison.
• Normalized claim-family SUBMTG_STATE_CD via trim(upper(...)) and aliased PYMT_OR_RCPMT_DT to PMT_OR_RCPMT_DT for joins.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (2)

taf/FTX/FTX.py:60

• The de-dup key groups by the raw PMT_OR_RCPMT_DT, but the join back to the base table compares payment dates using coalesce(..., to_date('1960-01-01')). If the source data mixes NULLs and the sentinel 1960-01-01 for the same claim key, both groups can pass count(...) = 1, and the coalescing join will match them as equal (causing duplicate rows / row multiplication). Make the de-dup key use the same null-normalized expression as the join predicate so NULL and sentinel are treated consistently.

                        ,SUBMTG_STATE_CD
                        ,ORGNL_CLM_NUM
                        ,ADJSTMT_CLM_NUM
                        ,ADJSTMT_IND
                        ,PMT_OR_RCPMT_DT

taf/FTX/FTX.py:96

• CLM_FMLY_{_2x_segment} groups by the raw PYMT_OR_RCPMT_DT (aliased to PMT_OR_RCPMT_DT), but the join to FTX uses a null-normalized comparison via coalesce(..., to_date('1960-01-01')). If the claim family table contains both NULL and the sentinel date for the same claim key, the CTE can keep multiple rows (each count(...) = 1) that later match as equal in the join, producing duplicates. Coalesce the claim-family payment date in the CTE so grouping and join semantics are aligned.

                            ,trim(upper(SUBMTG_STATE_CD)) as SUBMTG_STATE_CD
                            ,COALESCE(UPPER(ADJSTMT_IND),'X') as ADJSTMT_IND
                            ,PYMT_OR_RCPMT_DT as PMT_OR_RCPMT_DT