Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 68 additions & 23 deletions deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1178,35 +1178,80 @@ failures and scope limits; functional production validation is not a maximum-cap

## Incremental-sync page index

Migration 019 adds `idx_servers_updated_page_cover` on `(server_name, version)`
including `updated_at`, `is_latest`, and `status`. Unlike the partial search
index, it includes deleted versions required by incremental sync. Incremental
queries select a page of keys before fetching payloads. They retain prepared
statement reuse, binary payload transfer, and PostgreSQL automatic plan selection.
Forcing unnamed text-format execution regressed concurrent throughput in staging. The
watermark comparison, cursor ordering, filters, and response format are unchanged.

On a busy existing database, prebuild before deploying the application:
Migration 020 replaces migration 019's included-column timestamp index with
`idx_servers_updated_name_cover` on `(server_name, version, updated_at)` including
`is_latest` and `status`. It also replaces the single-column timestamp index with
`idx_servers_updated_time_cover` on `(updated_at DESC, server_name, version)`.
Both include deleted versions needed by incremental-sync clients.

For plain `updated_since` queries with `limit=100`, the query first probes up to
513 matching keys in timestamp order. With at most 512 matches it sorts that
complete set into name/version order; otherwise it fetches the requested page in
name order. Other incremental shapes without substring search retain the smaller
129-key probe and 128-match boundary. The larger probe avoids switching to a
costly name scan for moderately sparse pages. To avoid repeatedly paying for that
probe on large histories, each app samples one in 16 eligible plain 100-row
requests and retains up to 32 query-strategy hints for five seconds. A hint
records a timestamp/name range containing more than 512 matches. Requests
covered by that range can skip the probe; PostgreSQL verifies cursor ordering
behind a materialized guard so this decision does not defeat generic-plan reuse.
Hints are separated by deletion filtering and are not used in caller-owned
transactions or additional filter shapes. They contain no response data: writes
can make a hint suboptimal until it expires, but every request still applies the
complete predicate to current data. Sampled queries return an internal marker
row that is removed before pagination, serialization and returned-row metrics.
Both branches use the full predicate and one statement snapshot, with
payloads fetched only for the selected page. Timestamp order never becomes the
public pagination order. Plain incremental queries with page sizes 1 or 100 use
cached statements with the validated page size embedded in the query; cutoffs and
cursors remain parameters. The exact page bound lets PostgreSQL consider reusing
a generic plan instead of repeatedly planning the common shapes. Additional
filters and other page sizes retain per-execution planning. This is specific to
the adaptive query and replacement indexes, not a global plan-cache setting.
The probe bounds returned keys, not all index entries examined. Very large recent
histories, sparse cursors and different filter mixes still need measurement.

On a busy existing database, **prebuild both indexes before deploying** the
application. These steps assume migration 019 is already applied; older
deployments must also prebuild indexes for their earlier pending migrations.
Startup migrations run inside a transaction and cannot use
`CONCURRENTLY`; without a prebuild they use regular index creation, which blocks
writes during the build. Use a maintenance window if prebuilding is not possible.
Run these as separate statements in autocommit, outside a transaction:

```sql
SET application_name = 'registry-updated-page-index-build';
SET application_name = 'registry-updated-listing-index-build';
SET lock_timeout = '5s';
SET statement_timeout = '10min';
CREATE INDEX CONCURRENTLY idx_servers_updated_page_cover
ON public.servers (server_name, version) INCLUDE (updated_at, is_latest, status);
CREATE INDEX CONCURRENTLY idx_servers_updated_name_cover
ON public.servers (server_name, version, updated_at) INCLUDE (is_latest, status);
CREATE INDEX CONCURRENTLY idx_servers_updated_time_cover
ON public.servers (updated_at DESC, server_name, version);

SELECT c.relname, i.indisvalid, i.indisready, pg_get_indexdef(i.indexrelid)
FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname IN ('idx_servers_updated_name_cover', 'idx_servers_updated_time_cover');
```

Run in autocommit, outside a transaction. Verify the definition and both
`indisvalid` and `indisready` before deploying. The migration creates the index
on fresh databases and rejects an existing invalid index. A concurrent build
uses CPU, I/O, and disk space; monitor it and repair any incomplete build before
retrying.

Compare recent, old, and empty incremental-sync windows, deep cursors, deleted
records, and latest/version filters. Measure SQL execution and buffer access as
well as end-to-end latency, pool queues, CPU, and actual completed request rate.
Rolling back the application can leave this additive index in place; removal
is optional and should happen only after all candidate instances are gone.
Check both definitions and both readiness flags before deployment. Concurrent
builds consume CPU, I/O and temporary disk space; the old indexes remain until
migration 020 succeeds. If a build fails, inspect its state, drop only that new
incomplete index with `DROP INDEX CONCURRENTLY`, and retry in autocommit. Do not
use `IF NOT EXISTS` to assume a failed prebuild is usable.

Migration 020 validates the replacements, then drops `idx_servers_updated_page_cover`
and `idx_servers_updated_at` in the same transaction. Its five-second lock timeout
bounds lock acquisition, not the duration of an index build. A failure rolls back
the migration and retains the old indexes; investigate before retrying. It does
not change table data, connection limits, HPA targets or replica counts.

An application rollback remains schema-compatible with the replacement indexes,
but its planner choices and performance may differ. Leave the replacements in
place during an application rollback and measure behavior; restoring the exact
old physical layout requires a separate concurrent prebuild and reviewed cleanup.
Validate sparse/empty windows, early/deep cursors, `limit=1`, publishing, deleted
records and latest/version/search filters before production deployment. Compare
completed request rate, latency, CPU and pool queues at matched workloads.

### Packed production workers

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
-- Prebuild both indexes CONCURRENTLY on busy databases before deployment.
-- Startup migrations are transactional; see deploy/README.md for the prebuild.
-- A new name avoids silently reusing 019's included-column timestamp layout.
SET LOCAL lock_timeout = '5s';

CREATE INDEX IF NOT EXISTS idx_servers_updated_name_cover
ON servers (server_name, version, updated_at) INCLUDE (is_latest, status);
CREATE INDEX IF NOT EXISTS idx_servers_updated_time_cover
ON servers (updated_at DESC, server_name, version);

DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_index
WHERE indexrelid = 'idx_servers_updated_name_cover'::regclass
AND indrelid = 'servers'::regclass
AND indisvalid AND indisready
AND indnkeyatts = 3 AND indnatts = 5
AND indoption::text = '0 0 0'
AND indexprs IS NULL AND indpred IS NULL
AND (SELECT am.amname FROM pg_class c JOIN pg_am am ON am.oid=c.relam WHERE c.oid=indexrelid) = 'btree'
AND pg_get_indexdef(indexrelid, 1, true) = 'server_name'
AND pg_get_indexdef(indexrelid, 2, true) = 'version'
AND pg_get_indexdef(indexrelid, 3, true) = 'updated_at'
AND pg_get_indexdef(indexrelid, 4, true) = 'is_latest'
AND pg_get_indexdef(indexrelid, 5, true) = 'status'
) THEN
RAISE EXCEPTION 'idx_servers_updated_name_cover must be valid, ready, and have the expected definition; repair the prebuild before retrying migration 020';
END IF;
IF NOT EXISTS (
SELECT 1 FROM pg_index
WHERE indexrelid = 'idx_servers_updated_time_cover'::regclass
AND indrelid = 'servers'::regclass
AND indisvalid AND indisready
AND indnkeyatts = 3 AND indnatts = 3
AND indoption::text = '3 0 0'
AND indexprs IS NULL AND indpred IS NULL
AND (SELECT am.amname FROM pg_class c JOIN pg_am am ON am.oid=c.relam WHERE c.oid=indexrelid) = 'btree'
AND pg_get_indexdef(indexrelid, 1, true) = 'updated_at'
AND pg_get_indexdef(indexrelid, 2, true) = 'server_name'
AND pg_get_indexdef(indexrelid, 3, true) = 'version'
) THEN
RAISE EXCEPTION 'idx_servers_updated_time_cover must be valid, ready, and have the expected definition; repair the prebuild before retrying migration 020';
END IF;
END
$$;

-- Both replacements are usable before the old indexes are removed. The
-- transaction rolls back on lock timeout, retaining the old indexes on failure.
DROP INDEX IF EXISTS idx_servers_updated_page_cover;
DROP INDEX IF EXISTS idx_servers_updated_at;
82 changes: 51 additions & 31 deletions internal/database/postgres.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,10 @@ import (

// PostgreSQL is an implementation of the Database interface using PostgreSQL
type PostgreSQL struct {
pool *pgxpool.Pool
poolMetrics metric.Registration
listingMetrics metric.Int64Counter
updatedPageHints updatedPageHints
pool *pgxpool.Pool
poolMetrics metric.Registration
listingMetrics metric.Int64Counter
}

// Executor is an interface for executing queries (satisfied by both pgx.Tx and pgxpool.Pool)
Expand Down Expand Up @@ -220,28 +221,22 @@ func addCursorCondition(cursor string, argIndex int) (string, []any, int) {
}

// filteredPageQuery fetches payloads only after selecting a filtered page's keys.
func filteredPageQuery(filter *ServerFilter, ordinary, whereClause string, limitArg int, prepared ...bool) string {
func filteredPageQuery(filter *ServerFilter, ordinary, whereClause string, limitExpression string, prepared bool, strategy updatedPageStrategy) string {
if filter == nil || (filter.SubstringName == nil && filter.UpdatedSince == nil) {
return ordinary
}
// Find the page using only its keys before fetching the JSON payloads.
// The covering index can reject nonmatching names without visiting the
// heap. The materialized candidate query plans for a larger key batch:
// a small LIMIT otherwise assumes rare matches occur early in name order
// and can favor scanning most of the ordered index over a selective GIN
// lookup. The outer LIMIT still defines the exact page before payload fetch.
// PostgreSQL 17 can stop the ordered CTE scan after the requested rows.
// Select the page using only its keys before fetching JSON payloads.
payload := "s.value"
if len(prepared) > 0 && prepared[0] {
if prepared {
payload = listingPayload("s.")
}
// Incremental sync needs only the exact page. Its all-status covering index
// includes deleted versions, which consumers need to remove local records.
// Retain the larger planning boundary only for substring searches.
candidateLimit := fmt.Sprintf("$%d", limitArg)
if filter.SubstringName != nil {
candidateLimit = fmt.Sprintf("GREATEST($%d, 8192)", limitArg)
if filter.UpdatedSince != nil && filter.SubstringName == nil {
return updatedPageQuery(whereClause, limitExpression, payload, prepared, strategy)
}
// For substring searches, a small LIMIT can favor scanning most of the
// ordered index over a selective GIN lookup. Plan for a larger key batch;
// the outer LIMIT still defines the exact page before payload fetch.
candidateLimit := fmt.Sprintf("GREATEST(%s, 8192)", limitExpression)
return fmt.Sprintf(`
WITH candidates AS MATERIALIZED (
SELECT server_name, version FROM servers
Expand All @@ -254,18 +249,26 @@ func filteredPageQuery(filter *ServerFilter, ordinary, whereClause string, limit
FROM (
SELECT server_name, version FROM candidates
ORDER BY server_name, version
LIMIT $%d
LIMIT %s
) page
JOIN servers s USING (server_name, version)
ORDER BY s.server_name, s.version
`, whereClause, candidateLimit, payload, limitArg)
`, whereClause, candidateLimit, payload, limitExpression)
}

// cacheUpdatedPage bounds the extra statement-cache shapes to the two common
// page sizes. Other incremental filters retain parameter-specific planning.
func cacheUpdatedPage(filter *ServerFilter, limit int) bool {
return filter != nil && filter.UpdatedSince != nil &&
filter.SubstringName == nil && filter.Name == nil && filter.Version == nil &&
filter.IsLatest == nil && filter.RemoteURL == nil && (limit == 1 || limit == 100)
}

func filteredQueryArgs(filter *ServerFilter, args []any) []any {
if filter != nil && filter.SubstringName != nil {
// Search selectivity varies by orders of magnitude. Use an unnamed
if filter != nil && (filter.SubstringName != nil || filter.UpdatedSince != nil) {
// Search and incremental-sync selectivity vary greatly. Use an unnamed
// extended-protocol query so PostgreSQL plans for these parameter values
// instead of reusing one generic plan for every term.
// instead of reusing one generic plan for every term, cutoff, and cursor.
return append([]any{pgx.QueryExecModeExec}, args...)
}
return args
Expand Down Expand Up @@ -334,6 +337,7 @@ func (db *PostgreSQL) listServerRows(
whereConditions, args, argIndex := buildFilterConditions(filter, argIndex)

// Add cursor pagination
cursorArgIndex := argIndex
cursorCondition, cursorArgs, argIndex := addCursorCondition(cursor, argIndex)
if cursorCondition != "" {
whereConditions = append(whereConditions, cursorCondition)
Expand All @@ -351,22 +355,35 @@ func (db *PostgreSQL) listServerRows(
if prepared {
payload = listingPayload("")
}
cachePage := cacheUpdatedPage(filter, limit)
limitExpression := fmt.Sprintf("$%d", argIndex)
if cachePage {
// Only the validated constants 1 and 100 are embedded. Cutoffs and
// cursors remain parameters. This gives generic plans an exact page
// bound without forcing generic planning for other query shapes.
limitExpression = fmt.Sprintf("%d", limit)
}
query := fmt.Sprintf(`
SELECT server_name, version, status, status_changed_at, status_message, published_at, updated_at, is_latest, %s
FROM servers
%s
ORDER BY server_name, version
LIMIT $%d
`, payload, whereClause, argIndex)
query = filteredPageQuery(filter, query, whereClause, argIndex, prepared)
args = append(args, limit)
args = filteredQueryArgs(filter, args)
LIMIT %s
`, payload, whereClause, limitExpression)
strategy, args := db.chooseUpdatedPageStrategy(tx, filter, limit, cursorArgs, cursorArgIndex, argIndex, args)
query = filteredPageQuery(filter, query, whereClause, limitExpression, prepared, strategy)
if cachePage {
args = append([]any{pgx.QueryExecModeCacheStatement}, args...)
} else {
args = append(args, limit)
args = filteredQueryArgs(filter, args)
}

rows, err := db.getExecutor(tx).Query(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("failed to query servers: %w", err)
}
return readListingRows(ctx, rows, diagnostic, prepared)
return db.readUpdatedPageRows(ctx, rows, diagnostic, prepared, strategy, filter)
}

type listingRow struct {
Expand All @@ -376,7 +393,7 @@ type listingRow struct {
prepared bool
}

func readListingRows(ctx context.Context, rows pgx.Rows, diagnostic *telemetry.ListingDiagnostic, prepared bool) ([]listingRow, error) {
func readListingRows(ctx context.Context, rows pgx.Rows, diagnostic *telemetry.ListingDiagnostic, prepared bool, observation *updatedPageObservation) ([]listingRow, error) {
defer rows.Close()

// Copy the bounded page before decoding JSON. Exhausting/closing pooled rows
Expand All @@ -386,7 +403,6 @@ func readListingRows(ctx context.Context, rows pgx.Rows, diagnostic *telemetry.L
var page []listingRow
diagnostic.Phase("row_fetch")
for rows.Next() {
diagnostic.RowReceived()
diagnostic.Phase("row_fetch")
if err := ctx.Err(); err != nil {
return nil, err
Expand All @@ -403,6 +419,10 @@ func readListingRows(ctx context.Context, rows pgx.Rows, diagnostic *telemetry.L
if err := rows.Scan(dest...); err != nil {
return nil, fmt.Errorf("failed to scan server row: %w", err)
}
if observation != nil && observation.consume(row) {
continue
}
diagnostic.RowReceived()
meta.Status = model.Status(status)
diagnostic.AddBytes(len(row.value))
page = append(page, row)
Expand Down
80 changes: 80 additions & 0 deletions internal/database/updated_indexes_internal_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
package database

import (
"context"
"testing"

"github.com/jackc/pgx/v5"
"github.com/stretchr/testify/require"
)

func TestUpdatedListingIndexMigration(t *testing.T) {
for _, mode := range []string{"fresh", "upgrade", "prebuilt", "wrong-definition"} {
t.Run(mode, func(t *testing.T) {
db := NewTestDB(t).(*PostgreSQL)
ctx := context.Background()
conn, err := db.pool.Acquire(ctx)
require.NoError(t, err)
defer conn.Release()
migrator := NewMigrator(conn.Conn())
migrations, err := migrator.loadMigrations()
require.NoError(t, err)
var migration Migration
for _, candidate := range migrations {
if candidate.Version == 20 {
migration = candidate
}
}
require.Equal(t, 20, migration.Version)
if mode != "fresh" {
_, err = conn.Exec(ctx, `
DROP INDEX idx_servers_updated_name_cover;
DROP INDEX idx_servers_updated_time_cover;
CREATE INDEX idx_servers_updated_page_cover ON servers(server_name,version) INCLUDE(updated_at,is_latest,status);
CREATE INDEX idx_servers_updated_at ON servers(updated_at DESC);
DELETE FROM schema_migrations WHERE version=20;
`)
require.NoError(t, err)
switch mode {
case "prebuilt":
// Concurrent builds must be separate autocommit statements.
_, err = conn.Exec(ctx, `CREATE INDEX CONCURRENTLY idx_servers_updated_name_cover ON servers(server_name,version,updated_at) INCLUDE(is_latest,status)`)
require.NoError(t, err)
_, err = conn.Exec(ctx, `CREATE INDEX CONCURRENTLY idx_servers_updated_time_cover ON servers(updated_at DESC,server_name,version)`)
require.NoError(t, err)
case "wrong-definition":
// The old INCLUDE layout must not silently satisfy the new index name.
_, err = conn.Exec(ctx, `CREATE INDEX idx_servers_updated_name_cover ON servers(server_name,version) INCLUDE(updated_at,is_latest,status)`)
require.NoError(t, err)
}
err = migrator.applyMigration(ctx, migration)
if mode == "wrong-definition" {
require.ErrorContains(t, err, "expected definition")
assertUpdatedIndexExists(t, conn.Conn(), "idx_servers_updated_page_cover", true)
assertUpdatedIndexExists(t, conn.Conn(), "idx_servers_updated_at", true)
assertUpdatedIndexExists(t, conn.Conn(), "idx_servers_updated_time_cover", false)
var applied bool
require.NoError(t, conn.QueryRow(ctx, "SELECT EXISTS(SELECT 1 FROM schema_migrations WHERE version=20)").Scan(&applied))
require.False(t, applied)
return
}
require.NoError(t, err)
}
assertUpdatedIndexExists(t, conn.Conn(), "idx_servers_updated_page_cover", false)
assertUpdatedIndexExists(t, conn.Conn(), "idx_servers_updated_at", false)
for _, name := range []string{"idx_servers_updated_name_cover", "idx_servers_updated_time_cover"} {
var usable bool
require.NoError(t, conn.QueryRow(ctx, `SELECT indisvalid AND indisready AND indnkeyatts=3 FROM pg_index WHERE indexrelid=to_regclass($1)`, name).Scan(&usable))
require.True(t, usable)
}
require.NoError(t, migrator.Migrate(ctx))
})
}
}

func assertUpdatedIndexExists(t *testing.T, conn *pgx.Conn, name string, expected bool) {
t.Helper()
var exists bool
require.NoError(t, conn.QueryRow(context.Background(), "SELECT to_regclass($1) IS NOT NULL", name).Scan(&exists))
require.Equal(t, expected, exists, name)
}
Loading
Loading