Skip to content

[DQT] Fix: DQT Beta Data Inconsistency#10348

Open
arnav-makkar wants to merge 6 commits intoaces:mainfrom
arnav-makkar:fix/9837-data-inconsistency
Open

[DQT] Fix: DQT Beta Data Inconsistency#10348
arnav-makkar wants to merge 6 commits intoaces:mainfrom
arnav-makkar:fix/9837-data-inconsistency

Conversation

@arnav-makkar
Copy link
Contributor

Brief summary of changes

The beta Data Query Tool was displaying candidates with no instrument data, while the old DQT correctly filtered them out.

Root Cause

  • Old DQT: Uses CouchDB which only returns documents that exist (candidates with data)
  • Beta DQT: Queries MySQL directly, returning all active candidates regardless of whether they have instrument data

When queries included both candidate metadata fields (PSCID, CandID) and instrument fields (BMI data), candidates with only metadata but no instrument data were appearing as empty rows.

Solution

Added filtering logic in QueryDataProvisioner._mergeIterators() to skip candidates where all non-metadata fields are empty. This mimics the old DQT's CouchDB behavior.

The filter excludes candidate metadata fields (PSCID, CandID, DoB, Sex) when checking for data presence, ensuring only candidates with actual instrument data are included in results.

Testing

  • Run a query with BMI instrument fields
  • Verify candidates DCC211 and DCC355 no longer appear in results
  • Verify candidates with actual BMI data are still displayed correctly

Before:
data-2026-02-07T18_59_19.122Z.csv

After:
data-2026-02-07T18_57_54.273Z.csv

Link(s) to related issue(s)

@github-actions github-actions bot added Language: PHP PR or issue that update PHP code Module: dataquery PR or issue related to (new) dataquery module labels Feb 7, 2026
@ridz1208
Copy link
Collaborator

ridz1208 commented Feb 8, 2026

See comment on linked issue

@arnav-makkar
Copy link
Contributor Author

I've updated the logic to use a frontend toggle instead of backend filtering, as suggested by @ridz1208.

  • Inline Mode: Shows two toggles ("Display candidates with no data" & "Display empty visits") because candidates can have partial data.
  • Other Modes (Cross-sectional, Longitudinal, Raw): Shows only one toggle ("Display candidates with no data") because row filtering sufficiently handles empty data (e.g., Cross-sectional treats each visit as a row, so candidate-level filtering is redundant).
  • Fixed a React Not FoundError when the result set is empty by manually rendering the "No result found" message.
  • Downloaded CSVs are consistent with the current frontend filters.

Testing:

  • Verified toggle behavior in all modes.
  • "No result found" displays correctly for 0 results without crashing.

Demo Videos:

Screen.Recording.2026-02-08.at.16.37.07.mov
Screen.Recording.2026-02-08.at.16.39.52.mov

The failing test is unrelated to changes in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Language: PHP PR or issue that update PHP code Module: dataquery PR or issue related to (new) dataquery module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DQT] - Inconsistent data displayed between old DQT and new DQT

2 participants