Skip to content

feat(compat): add Knex compatibility layer for MySQL and PostgreSQL#174

Merged
jeremydaly merged 16 commits into
mainfrom
feat/knex-compat
Jun 9, 2026
Merged

feat(compat): add Knex compatibility layer for MySQL and PostgreSQL#174
jeremydaly merged 16 commits into
mainfrom
feat/knex-compat

Conversation

@jeremydaly

@jeremydaly jeremydaly commented Jun 8, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a Knex compatibility layer so the Data API Client can back Knex for both MySQL and PostgreSQL, closing the last gap among the supported query builders (Drizzle and Kysely already work). CRUD, the common query-builder surface, transactions, and upserts all work.

  • Knex doesn't accept an injected pool like Drizzle/Kysely; it builds its own driver via Client._driver(). We subclass the mysql2 and pg dialects and override that single method to hand Knex a Data API-backed connection. Knex keeps full ownership of SQL generation; queries execute over the Data API.
  • New data-api-client/compat/knex subpath exporting createKnexMySQLClient(config) and createKnexPgClient(config). knex is lazy-required and declared as an optional peer dependency (^3.0.0).
  • Transactions work: the compat layer intercepts the literal BEGIN/COMMIT/ROLLBACK SQL Knex issues (src/compat/transaction-sql.ts) and maps it to the Data API transaction lifecycle, threading transactionId onto subsequent queries on that connection. db.transaction() commits on success and rolls back on a thrown callback.
  • Query-builder coverage: 26 tests per engine exercise the common syntax from the Knex guide, including selects/distinct/pluck/first, the where family, joins, groupBy/having/aggregates, ordering/pagination, unions, subqueries, CTEs, insert/update/del, returning, increment/decrement, and onConflict().merge() upserts.

Bugs fixed along the way

These were latent in the shared compat layers and benefit non-Knex callers too:

  • compat/pg binding clobber: the query({ text, values }, cb) callback form left params defaulting to [], which overrode config.values in executeQuery, so parameterized queries using node-postgres's config-object form lost their bindings (bind message supplies 0 parameters).
  • compat/mysql2 callback parsing: query(config, undefined, cb) (the form Knex uses for bindings-less statements like BEGIN;) never picked up the third-argument callback, so the callback never fired and transactions hung. Fixed in all four query parsers.
  • compat/pg CTE command inference: inferCommand() keyed off the first SQL word, so WITH ... SELECT was tagged QUERY instead of SELECT. node-postgres-driven callers (Knex/Kysely/Drizzle) read result shape off command, so a CTE returned the raw response object instead of rows. Now resolves the primary verb at parenthesis depth 0.

Test Plan

  • Unit tests pass: npm run test:unit (157 passed)
  • Knex integration, both engines: npm run test:int:orm:knex (74 passed)
    • CRUD, transactions (commit/rollback/nested-rejection), and the full query-builder coverage
  • Regression: pg compat, Kysely, and Drizzle suites still pass (the shared compat/pg and compat/mysql2 fixes do not break existing callers)
  • tsc build emits dist/compat/knex.{js,d.ts} and dist/compat/transaction-sql.js; ESLint clean

Integration tests require .env.local (live Aurora Serverless v2 clusters). Declared engines.node >= 20 (.nvmrc of 20) to match CI and the vitest 4 toolchain floor.

Limitations

  • Nested transactions need SQL SAVEPOINTs, which the Data API has no primitive for, so a nested trx.transaction(...) throws a clear error. A single top-level transaction works.
  • Streaming (.stream()) is unsupported, since the Data API has no cursor API.
  • Knex's dedicated whereLike() appends COLLATE utf8_bin, which MySQL rejects on utf8mb4 columns. This is a Knex/MySQL quirk that affects real mysql2 too, not the compat layer; the portable where(col, 'like', val) form works.

Notes

  • The deep import knex/lib/dialects/{mysql2,postgres} is Knex's documented-but-internal extension point, so a Knex major version could move it, which is why the peer range is ^3.0.0.
  • This branch also carries a CLAUDE.md refresh, an npm audit fix for three dev-dependency advisories, and a note that .env.local must be sourced in the same shell command as the test run.

The file had drifted from the source after the compat-layer, retry, and
ORM work landed. Corrected/added:

- Default engine documented as 'pg' (was wrongly 'mysql'; client.ts:65)
- New sections: Driver Compatibility Layers (compat/pg, compat/mysql2,
  compat/errors) and Retry Behavior (scale-to-zero wake-up retries)
- Config block: added retryOptions and namedPlaceholders
- Testing: replaced dead test:integration* scripts with the real
  test:int:core / :compat / :orm:* suites; listed compat + ORM test files
- Dependencies bumped to match package.json (SDK 3.1048, vitest 4, @types/pg)
- Key Files tree: added retry.ts, src/compat/, new integration tests, infra/,
  and the package subpath exports
- Project Status: 2.0 beta -> 2.x stable with feature summary
Knex can't take an injected pool like Drizzle/Kysely — it builds its own
driver via Client._driver(). We subclass the mysql2 and pg dialects and
override _driver() to hand Knex a Data API-backed connection, so Knex
keeps full ownership of SQL generation while queries run over the Data API.

- src/compat/knex.ts: createKnexMySQLClient / createKnexPgClient
  (lazy-require knex so it stays an optional peer dependency)
- Export both from compat/index.ts; add ./compat/knex subpath export
- knex added as optional peerDependency (^3.0.0)
- Replace the all-skipped knex-mysql.int.test.ts with a real suite and
  add knex-pg.int.test.ts (8 live tests each: insert/returning, select,
  where, parameterized bindings, update, count, orderBy+limit, delete)
- test:int:orm:knex now runs both engines (+ :pg / :mysql variants)
- README: document Knex usage for both engines

Also fixes a latent compat/pg binding bug surfaced by Knex: the
query({ text, values }, cb) callback form left params defaulting to [],
which clobbered config.values in executeQuery — so parameterized queries
lost their bindings. Default params to undefined so config-embedded
values are honored (client and pool paths).

Limitation: Knex transactions (db.transaction()) are not supported — Knex
issues literal BEGIN/COMMIT/ROLLBACK SQL the Data API can't honor; use the
native client.transaction() instead. Documented in README and as skipped
tests.

Tests: 157 unit + 16 Knex live (2 skipped) + 102 pg compat/ORM regression
all pass. Requires Node 22 (repo's vitest 4).
The merged CLAUDE.md refresh predated the Knex compat layer and said
'Knex is NOT supported'. Update it to reflect compat/knex:
- Add a compat/knex bullet to Driver Compatibility Layers
- Rewrite ORM support status: Knex supported via _driver() override,
  both engines tested; keep the db.transaction() limitation note
- Fix Project Status and Testing suite lines
The repo's vitest 4 / rolldown toolchain requires Node 20.19+ (Node 18
fails on node:util styleText). CI integration tests already run on Node
20; codify that floor:
- engines.node: >=20 (consumer-facing; matches the lowest Node CI tests)
- .nvmrc: 20 (contributor default)
jeremydaly added 10 commits June 8, 2026 14:17
Resolves 3 transitive devDependency advisories (2 moderate, 1 high), all
ReDoS/hang issues in tooling deps (eslint/typescript-eslint chain):
- ajv 6.12.6 -> 6.15.0 (GHSA-2g4f-4pwh-qvx6)
- brace-expansion -> 1.1.15 / 2.1.1 (GHSA-v6h2-p8h4-qcjw, GHSA-f886-m6hf-6m8v)
- cross-spawn -> 7.0.6 (GHSA-3xgq-45jj-v275)

Dev-only; does not affect the published package (ships dist/ only).
package.json unchanged; npm audit now reports 0 vulnerabilities.
Note the shell-persistence gotcha: env vars from `source .env.local` do
not carry across separate commands, and nothing auto-loads the file, so
it must be chained (`source .env.local && npm run test:int:...`).
ORMs/Knex drive transactions by issuing literal BEGIN/COMMIT/ROLLBACK SQL
on the connection. Intercept and map these to the Data API transaction
lifecycle, threading transactionId onto subsequent queries.

- src/compat/transaction-sql.ts (new): classifyTransactionControl()
  normalizes a statement (strips trailing ';', collapses whitespace) and
  recognizes BEGIN/BEGIN TRANSACTION/START TRANSACTION, COMMIT, ROLLBACK,
  SAVEPOINT/RELEASE/ROLLBACK TO SAVEPOINT, and SET TRANSACTION.
- pg compat: replace the inline matcher (which used === 'BEGIN' and missed
  Knex's 'BEGIN;') with the shared classifier.
- mysql2 compat: add the same interception (it previously had only
  lifecycle methods, never intercepting the SQL strings).
- mysql2 compat: fix query() arg-parsing so query(config, undefined, cb)
  picks up the 3rd-arg callback. Knex sends this form for bindings-less
  transaction statements; missing it caused transactions to hang. Fixed in
  all four query parsers (connection + pool, object + string forms).
- Nested transactions need SAVEPOINTs, which the Data API has no primitive
  for, so the classifier throws a clear error; documented and tested.

Replace the skipped 'not supported' tests with real commit/rollback/nested
tests in both Knex suites (6 new). README and CLAUDE.md updated.

Tests: 22/22 Knex (both engines), 157 unit + 223 compat/ORM regression,
all pass.
…ence

Add query-builder coverage suites for both engines (26 tests each) exercising
the common syntax from knexjs.org/guide/query-builder.html: select/distinct/
pluck/first, the where family (whereIn/whereNull/whereBetween/whereExists/
whereRaw/...), joins, groupBy/having/aggregates, orderBy/limit/offset, unions,
subqueries, CTEs, insert/update/del, returning (pg), increment/decrement, and
onConflict().merge() upserts.

Fix surfaced by the CTE tests: inferCommand() returned 'QUERY' for statements
starting with WITH, so Knex's pg processResponse didn't treat WITH...SELECT as a
SELECT and returned the raw response instead of rows. Resolve the primary verb
at parenthesis depth 0 (CTE bodies are nested, so their verbs are skipped).
Benefits all pg compat callers doing CTEs, not just Knex.

Notes: Knex's whereLike() appends COLLATE utf8_bin which MySQL rejects on
utf8mb4 columns (a Knex/MySQL quirk, not a compat issue); the portable
where(col,'like',val) form is used. Streaming (.stream()) is unsupported.

test:int:orm:knex now runs the query-builder suites too. Tests: 74 Knex
(both engines) + 157 unit + pg regression all pass.
Minor bump for the new Knex compatibility layer (compat/knex) with
transaction support and query-builder coverage.
- Drop the outdated v2.1.0 'Note' callout; keep only the v1.x pointer at the top
- Remove the inline 'Version 2.1/2.0' intro paragraphs (superseded below)
- Add a 'What's New in v2.3' section for the Knex compat layer
- Collapse v2.2/v2.1/v2.0 into a consolidated Changelog
- Move Knex out of the v2.1 entry (it lands in v2.3)
Mention Knex, Drizzle, and Kysely by name in the opening paragraph, and
give it a bit more energy.
@jeremydaly jeremydaly marked this pull request as ready for review June 8, 2026 19:51
@jeremydaly jeremydaly merged commit b1e8100 into main Jun 9, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant