Skip to content

feat: add .sqlx (Dataform) file support #3496

Description

@DonTodt

Context

We use graphify to build knowledge graphs over a set of GCP data/analytics repos (Terraform infra + BigQuery pipelines). A good chunk of our SQL pipeline logic lives in Google Cloud Dataform, which uses .sqlx files rather than plain .sql.

Right now .sqlx isn't in the supported file type table, so these files are silently skipped during extract (same bucket as e.g. .gitignore — "no supported extension"). For repos where Dataform is the primary way SQL/pipeline logic is defined, that's a meaningful blind spot in the graph.

What a .sqlx file looks like

A Dataform .sqlx file is SQL plus a Dataform-specific config block and templating, e.g.:

config {
  type: "table",
  schema: "analytics",
  tags: ["daily"],
  dependencies: ["stg_events"]
}

js {
  const cutoffDays = 30;
}

SELECT
  user_id,
  COUNT(*) AS events
FROM ${ref("stg_events")}
WHERE event_date >= DATE_SUB(CURRENT_DATE(), INTERVAL ${cutoffDays} DAY)
GROUP BY user_id

This is why treating .sqlx as plain SQL (even with the [sql] extra / tree-sitter-sql) doesn't really work — the config {} block and ${ref(...)} / js {} templating aren't valid SQL syntax and would likely break or silently produce nothing useful.

What would be useful to extract

  • ref("other_model") / ${ref(...)} calls → depends_on edges between Dataform models, similar to how package manifests currently get a canonical node + depends_on edges
  • The config {} block metadata (type, schema, tags, dependencies) as node attributes
  • Treating each .sqlx file as a node representing a Dataform model/table, so it shows up in the graph the same way a .py/.tf file does

Even a first pass that just parses the config {} block and ref()/self() calls (without fully understanding the SQL body) would already make Dataform-heavy repos much more useful to query — that's the same value proposition as the existing [terraform] extra for .tf files.

Suggested approach

Given the pattern already used for other niche formats, an opt-in extra along the same lines as [terraform]/[sql] would make sense, e.g.:

uv tool install "graphifyy[dataform]"

with .sqlx added to the file type table.

Happy to test against a real-world corpus of .sqlx files if that's useful for building/validating an extractor.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions