Fixes #27158: ingestion slowdown from tag_usage seq-scan on Postgres

[sonika-shah]

Fixes #27158

Summary

getTagsInternalByPrefix parallel seq-scans tag_usage on Postgres, causing RDS CPU spikes during ingestion.

Cause: 1.11.0 (#23054) added four partial indexes on tag_usage filtered WHERE state = 1. #24063 then dropped the matching state = 1 filter from the query (Suggested rows are valid for both classification and glossary derivation), leaving every partial index inapplicable. MySQL was unaffected — its 1.11.0 indexes were never partial (no partial-index syntax).

Fix

bootstrap/sql/migrations/native/1.12.7/postgres/schemaChanges.sql:

1. Add non-partial single-col btrees on targetfqnhash_lower and tagfqn_lower — mirrors MySQL's idx_targetfqnhash_lower / idx_tagfqn_lower from 1.11.0.
2. Rebuild the four 1.11.0 partials as non-partial — same shape, same INCLUDE columns; only WHERE state = 1 removed so future predicate changes can't silently invalidate them.

All DDL is CONCURRENTLY and idempotent. No Java/query change. No MySQL change needed.

Verification

50k synthetic rows in local Postgres:

	Plan	Buffers
Before	Seq Scan (Rows Removed by Filter: 49010)	2274
After	Bitmap Index Scan on idx_tag_usage_targetfqnhash_lower_pattern	1024
Prepared / generic plan	Index still picked; LIKE LOWER($1) → range scan	24 (index only)

Test plan

• Reproduced seq scan on a representative dataset
• Verified bitmap index scan after fix (inline + prepared statement)
• Verified rebuilt composite still serves source-filtered queries
• Verified migration idempotency

---

Summary by Gitar

• Database schema:
  • Added GIN index gin_tag_usage_targetfqn_trgm using pg_trgm extension for optimized pattern searching.
• Dependency update:
  • Enabled pg_trgm extension on the database to support new GIN indexing.
• Miscellaneous:
  • Included a cherry-pick from PR fix(description-sanitizer): allow BlockEditor file attachment and callout attributes on div elements #27658 related to description-sanitizer that was not mentioned in the description.

_{This will update automatically on new commits.}

[Copilot]

Pull request overview

Restores efficient Postgres execution for tag_usage prefix-LIKE lookups by reintroducing a usable index for the current query shape and removing the brittle coupling between query predicates and partial index predicates.

Changes:

• Add a non-partial btree index on tag_usage.targetfqnhash_lower using text_pattern_ops to serve prefix LIKE queries.
• Rebuild the existing tag_usage partial indexes (previously WHERE state = 1) as non-partial indexes to avoid future predicate-coupling regressions.
• Rebuild the existing gin_tag_usage_targetfqn_trgm index without the partial predicate.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[github-actions]

🟡 Playwright Results — all passed (14 flaky)

✅ 3984 passed · ❌ 0 failed · 🟡 14 flaky · ⏭️ 86 skipped

Shard	Passed	Failed	Flaky	Skipped
🟡 Shard 1	297	0	2	4
🟡 Shard 2	749	0	5	8
🟡 Shard 3	745	0	1	7
🟡 Shard 4	774	0	1	18
🟡 Shard 5	686	0	1	41
🟡 Shard 6	733	0	4	8

🟡 14 flaky test(s) (passed on retry)

• Features/DataAssetRulesDisabled.spec.ts › Verify the Messaging Service entity item action after rules disabled (shard 1, 1 retry)
• Features/TeamsHierarchy.spec.ts › Delete Parent Team (shard 1, 1 retry)
• Features/ActivityAPI.spec.ts › Activity event is created when description is updated (shard 2, 1 retry)
• Features/ActivityAPI.spec.ts › Activity event is created when owner is added (shard 2, 1 retry)
• Features/DataQuality/TestCaseImportExportE2eFlow.spec.ts › Admin: Complete export-import-validate flow (shard 2, 1 retry)
• Features/DataQuality/TestCaseResultPermissions.spec.ts › User with only VIEW cannot PATCH results (shard 2, 1 retry)
• Features/Glossary/GlossaryWorkflow.spec.ts › should display correct status badge color and icon (shard 2, 1 retry)
• Features/RTL.spec.ts › Verify Following widget functionality (shard 3, 1 retry)
• Pages/Entity.spec.ts › Tier Add, Update and Remove (shard 4, 1 retry)
• Pages/ExplorePageRightPanel.spec.ts › Should verify deleted user not visible in owner selection for table (shard 5, 1 retry)
• Pages/Glossary.spec.ts › Async Delete - WebSocket failure triggers recovery (shard 6, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: searchIndex (shard 6, 1 retry)
• Pages/ODCSImportExport.spec.ts › Multi-object ODCS contract - object selector shows all schema objects (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[gitar-bot]

Code Review ✅ Approved 2 resolved / 2 findings

Adds a GIN trigram index on the tag_usage column to resolve ingestion slowdowns caused by sequential scans. This change addresses previously identified index targeting and privilege concerns.

✅ 2 resolved

✅ Performance: GIN trigram index targets wrong column for the slow query

📄 bootstrap/sql/migrations/native/1.12.7/postgres/schemaChanges.sql:39-42
The gin_tag_usage_targetfqn_trgm index (line 41-42) is built on targetFQNHash (original-case column), but getTagsInternalByPrefix filters on targetfqnhash_lower with a prefix LIKE. This index won't serve the problematic query. Additionally, prefix LIKE is already served by the new text_pattern_ops btree index at line 18-19, making this GIN trigram index redundant for the stated fix.

If there is a future use-case for infix LIKE '%…%' searches on targetFQNHash, the trigram index makes sense — but that should be noted in the migration comment. As-is, it adds write overhead (GIN maintenance on every insert/update) with no query benefit for the fix in this PR.

✅ Edge Case: CREATE EXTENSION pg_trgm may fail without superuser privileges

📄 bootstrap/sql/migrations/native/1.12.7/postgres/schemaChanges.sql:39
CREATE EXTENSION IF NOT EXISTS pg_trgm (line 39) requires the CREATE privilege on the database and, depending on PostgreSQL configuration, may need superuser or rds_superuser role on managed services like RDS. If the migration user lacks this privilege, the entire migration will fail — blocking the critical btree index creation that precedes it.

Consider moving the extension creation + GIN index to the end of the file (or a separate optional script) so the core fix (the text_pattern_ops indexes) isn't gated on extension availability.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}