[bpepple]

Summary

Adds near-match cover hash search to the issue API, so clients can identify an issue from a cover image even when the hash isn't an exact match:

/api/issue/?cover_hash_near=<16 hex chars>&cover_hash_max_distance=<0-20>

• Indexed cover_hash_bits column: a stored generated bigint that PostgreSQL computes from cover_hash with a HexToBigint expression (NULL when the hash is blank or malformed), with a partial covering index (issue_cover_hash_bits_idx). Migration 0062 adds both.
• CoverHashDistance DB expression: computes the Hamming distance between two of those bigints in PostgreSQL as bit_count((a # b)::bit(64)) (PostgreSQL 14+). The query hash is converted with the same HexToBigint expression as the stored column, so the two can't disagree. PostgreSQL folds it to a constant once per query. One PHASH_REGEX constant defines a valid hash for both the generated column and the filter's input validation.
• cover_hash_near filter: returns issues whose main cover hash is within cover_hash_max_distance bits (default 10, max 20), sorted closest first. It finds matching ids with an index-only scan of issue_cover_hash_bits_idx, then fetches those issues by primary key. Issues without a valid cover hash are excluded, so one malformed stored value can't break the search. Invalid hashes or distances return 400 Bad Request. It can be combined with the existing issue filters.
• No caching for near-match queries: submitted hashes are practically unique, so their responses would almost never be reused. A new uncached_list_params hook on CachedListModelMixin bypasses the list cache whenever cover_hash_near is given. A whitespace-only value counts as not given, matching how the filter treats it.
• Docs: the API README and the IssueViewSet docstring describe both parameters, how to compute a compatible hash with imagehash.phash(), and how to retry with a wider distance when the default finds nothing.

Choosing the default distance

The default is set for the main users: people tagging digital comic files. I tested 1,271 MetronInfo-tagged CBZs from three series (Captain America, Batman, and Something Is Killing the Children) against all ~184k hashed issues:

Max distance	Correct issue found	Correct issue is the only result	Has a wrong match	Results (median / max)
8	97.6%	96.2%	1.3%	1 / 3
10	98.6%	92.1%	6.6%	1 / 5
12	99.2%	61.1%	38%	1 / 18
16	99.7%	0%	100%	37 / 329

Each series on its own gave similar numbers, and 10 was the best balance for all three. Going to 12 gains less than 1% recall while the share of queries that return a wrong match jumps from 7% to 38%.

What the misses turned out to be:

• Print vs. digital trade dress: most issues more than 10 bits off came from Batman (2016). I compared the Metron images for the ten worst (12–18 bits) with the digital files. The art is the same, but Metron has the print covers, with a price box and UPC barcode, and the digital editions drop those. The Metron images are correct, so the README now suggests retrying with a larger cover_hash_max_distance when the default finds nothing, and treating the results as candidates.
• Different cover version on Metron: Something Is Killing the Children #31 is 18 bits off because Metron's image has a different-coloured figure from the published cover, probably the solicitation image. That cover was replaced.
• Wraparound covers: Captain America (2018) #1's file uses the double-page spread (back cover on the left, front on the right) as its cover, which is 24 bits off, beyond the maximum distance. Splitting it at the midpoint, as Metron-Tagger does (Match double-page spread covers against single Metron covers metron-tagger#288), puts the right half at 0 bits and makes it the only result. The README now describes this for client authors.

Two limits that no threshold can fix:

• Unrelated covers can be as close as 8 bits.
• Collected editions often reuse an issue's cover exactly. For example, Something Is Killing the Children #17 matched *TPB* #4 at 0 bits, ahead of the issue itself at 2.

Narrowing wider searches with other filters

cover_hash_near can be combined with the existing issue filters, which makes a wider retry much more usable. If the client knows the cover year (from the filename or existing metadata), adding cover_year cuts the candidates sharply. Results across the same 1,271 files:

Max distance	Results (median / max)	With cover_year (median / 95th pct / max)	Correct issue is the only result, with cover_year
10	1 / 5	1 / 1 / 2	97.6%
12	1 / 18	1 / 1 / 3	96.8%
16	37 / 329	1 / 4 / 12	50.8%
24 (not allowed: above the max)	11,130 / 19,965	169 / 342 / 494	0%

For example: /api/issue/?cover_hash_near=<hash>&cover_hash_max_distance=16&cover_year=2018. Filters such as series_name or publisher_id narrow results the same way. Cover years can differ from the year in a filename (which is often the release year), so a client may want to try the adjacent year too.

So results should always be checked against the scanned cover. Camera scans are noisier (typically 6–16 bits off, per discussion #680), so callers matching photos can raise cover_hash_max_distance up to 20.

Only main covers are searched. Variants have no cover_hash yet.

Performance

Metron-Tagger users tagging whole collections will send one uncached query per file, so the query has to be cheap. Measured on a copy of production data (~184k hashed issues), DB time per request:

Max distance	Results (median / p90 / max)	Time per request (median / p90 / max)
10 (default)	1	~20 ms
16	36 / 69 / 91	21 / 23 / 25 ms
18	212 / 337 / 497	27 / 29 / 32 ms
20 (max)	979 / 1,391 / 1,923	36 / 45 / 78 ms
24 (for comparison)	11,064 / 13,273 / 15,663	130 / 183 / 197 ms

Distances 16–24 are 40 random stored hashes, each searched against the full corpus.

Before this change, the default-distance query took about 160 ms. The old query scanned the whole 84 MB issue table, and pagination's COUNT(*) ran that scan a second time. The new one reads about 8 MB of index, and the count only rechecks the matched ids. At large distances most of the time goes to sorting and paging the loose matches, not finding them.

Choosing the maximum distance

The maximum is 20. In @erickolb's camera testing (discussion #680), no top-8 match was ever more than 16 bits off, and he suggested 20 as an upper bound. At 24 a query matched about 11k covers, which no one would page through, at about 4× the cost of 20. 20 leaves 4 bits of headroom over the worst camera match seen. Raising the maximum later won't break clients, but lowering it would.

Deploying

• Adding the generated column rewrites comicsdb_issue, so the table is locked while the migration runs (about 4 seconds locally, including the index build).
• Run VACUUM ANALYZE comicsdb_issue; right after migrating. The rewritten table starts without the visibility data the index-only scan relies on. Searches still work until it's vacuumed, just more slowly.

Testing

• Tests for CoverHashDistance, and that the generated cover_hash_bits holds the expected signed 64-bit values (including hashes with the high bit set, stored as negative bigints, and uppercase hex) and is NULL for blank or malformed hashes.
• Tests for the cover_hash_near / cover_hash_max_distance filters: ordering, default and max distance, validation (including cover_hash_max_distance sent on its own), skipping malformed stored hashes, and combining with other filters. That includes upc, creator_id and role_id, which join to-many relations and add DISTINCT.
• Tests that near-match queries skip the cache and whitespace-only values don't.
• pytest tests/comicsdb/test_issue_cover_hash_filters.py tests/comicsdb/test_cover_hash_distance.py tests/comicsdb/test_api_response_caching.py: 84 passed. Full suite: 2,270 passed.