Skip to content

Latest commit

 

History

History
931 lines (691 loc) · 30.3 KB

migrations.md

File metadata and controls

931 lines (691 loc) · 30.3 KB

Migration Writing Guide

This document covers everything you need to write, read, and customise migration files for the morphic migration framework.

Migrations are auto-generated by morphic generate, but you will often need to:

  • Understand what a generated file does
  • Modify a generated migration (add a data backfill, change a field option)
  • Write a migration entirely by hand (complex DDL, RunSQL, data seeding)
  • Understand why a particular operation was generated

Runtime model. Migration files are Starlark (.star) source files — a Python-like DSL powered by Starlark-Go. At runtime morphic migrate loads each .star file, evaluates it in the Starlark-Go interpreter, and runs the migrations in the morphic process. Each file must call migration() exactly once at the top level. The Starlark dialect is deterministic and sandboxed — there are no imports, no side effects, and no filesystem access. For anything the typed builtins cannot express, use run_sql().


Anatomy of a Migration File

Every migration is a Starlark source file in the migrations/ directory. It calls migration() at the top level, which registers itself with the migration runner.

migration(
    name = "0002_add_phone",
    dependencies = ["0001_initial"],
    operations = [
        add_field("users", varchar("phone", 20, nullable=True)),
    ],
)

Key points:

  • Each .star file calls migration() exactly once
  • name — must be unique across all migrations; conventionally NNNN_description
  • dependencies — names of migrations that must be applied before this one
  • operations — ordered list of schema changes to apply

The migration() Function

migration(
    name = "0001_initial",           # Required: unique migration identifier
    dependencies = [],               # Optional: list of prerequisite migration names
    operations = [...],              # Optional: list of operations to apply
)
Parameter Type Required Description
name string Yes Unique identifier, e.g. "0001_initial"
dependencies list of strings No Names of migrations this depends on (default: [])
operations list of operations No Schema operations to apply, in order (default: [])

name

The unique identifier for this migration. The morphic generate command uses the format NNNN_description (e.g. 0003_add_user_preferences). Keep names:

  • Unique — duplicate names cause an error at startup
  • Descriptive — they appear in status output and the history table
  • Stable — once applied in production, never rename a migration

dependencies

Names of migrations that must be fully applied before this one runs. The DAG is built from these links. An empty list means the migration has no prerequisites.

# No dependencies (first migration)
migration(name="0001_initial", operations=[...])

# Single linear dependency
migration(name="0002_add_phone", dependencies=["0001_initial"], operations=[...])

# Merge migration: depends on two branch leaves
migration(name="0004_merge", dependencies=["0002_branch_a", "0003_branch_b"], operations=[])

operations

An ordered list of operation calls. They are applied top-to-bottom on up and reversed bottom-to-top on down. See All Operations below.


Field Types

Fields describe table columns. There are two ways to define a field: typed builtins (preferred) and the generic field() fallback.

Typed Field Builtins

Each type has a dedicated function with type-appropriate positional arguments:

Builtin Positional Args Example
uuid(name) name uuid("id", primary_key=True, default="new_uuid")
varchar(name, length) name, length varchar("email", 255)
text(name) name text("notes", nullable=True)
integer(name) name integer("sort_order", default="zero")
bigint(name) name bigint("total_bytes")
boolean(name) name boolean("is_active", default="true")
timestamp(name) name timestamp("created_date", default="now")
date(name) name date("due_date", nullable=True)
time(name) name time("start_time")
float(name) name float("weight", nullable=True)
decimal(name, precision, scale) name, precision, scale decimal("unit_price", 10, 2)
jsonb(name) name jsonb("metadata", default="object")
bytes(name) name bytes("file_data")
serial(name) name serial("seq_no")
foreign_key(name, fk) name, fk dict foreign_key("contact_id", fk("contact", on_delete="CASCADE"))

All typed builtins accept these common keyword arguments:

Keyword Type Default Description
nullable bool False Allow NULL values
primary_key bool False Mark as primary key
default string "" Symbolic default name (resolved at runtime via set_defaults)

Datetime types (timestamp, date, time) also accept:

Keyword Type Default Description
auto_create bool False Auto-set on row creation (e.g. created_at)
auto_update bool False Auto-set on row update (e.g. updated_at)

Generic Field Fallback

For column types without a dedicated builtin, use field():

field("name", "citext", nullable=True)
field("data", "hstore", default="blank")
Parameter Position Type Required Description
name 1st string Yes Column name
type 2nd string Yes SQL type name
nullable keyword bool No Allow NULL (default: False)
primary_key keyword bool No Primary key (default: False)
default keyword string No Symbolic default name
length keyword int No Max length (default: 0)
precision keyword int No Decimal precision (default: 0)
scale keyword int No Decimal scale (default: 0)
auto_create keyword bool No Auto-set on create (default: False)
auto_update keyword bool No Auto-set on update (default: False)
foreign_key keyword dict No FK reference from fk()
many_to_many keyword string No M2M target table name

Field Types Reference

The field type maps to the provider's SQL dialect. Common values:

Type PostgreSQL MySQL SQLite Notes
"integer" INTEGER INT INTEGER
"bigint" BIGINT BIGINT INTEGER
"varchar" VARCHAR(n) VARCHAR(n) TEXT Requires length
"text" TEXT TEXT TEXT
"boolean" BOOLEAN TINYINT(1) INTEGER
"decimal" DECIMAL(p,s) DECIMAL(p,s) REAL Requires precision/scale
"float" FLOAT FLOAT REAL
"timestamp" TIMESTAMP DATETIME TEXT
"date" DATE DATE TEXT
"uuid" UUID CHAR(36) TEXT
"jsonb" JSONB JSON TEXT
"bytes" BYTEA BLOB BLOB Binary data
"foreign_key" INTEGER + FK INTEGER + FK INTEGER + FK Pair with fk()

Foreign Key References

The fk() helper builds a foreign key reference dict:

fk("contact", on_delete="CASCADE")
fk("item", on_delete="SET_NULL", on_update="CASCADE")
Parameter Position Type Required Description
table 1st string Yes Referenced table name
on_delete keyword string No Delete action: CASCADE, PROTECT, SET_NULL, SET_DEFAULT, NO_ACTION, RESTRICT
on_update keyword string No Update action (same options as on_delete)

Use fk() with the foreign_key typed builtin:

foreign_key("created_by_id", fk("users", on_delete="PROTECT"), nullable=True)

Or with the generic field():

field("owner_id", "uuid", foreign_key=fk("users", on_delete="CASCADE"), nullable=True)

Indexes

The index() helper defines table indexes:

index("contact_email_idx", ["email"], unique=True)
index("job_contact_date_idx", ["contact_id", "due_date"])
index("item_metadata_idx", ["metadata"], method="GIN")
Parameter Position Type Required Description
name 1st string Yes Index name (must be unique in the database)
fields 2nd list of strings Yes Column names to index
unique keyword bool No Unique constraint (default: False)
method keyword string No Index method: BTREE, HASH, GIN, GiST, BRIN
where keyword string No Partial index WHERE clause

All Operations

create_table

Creates a new table with fields and optional indexes.

create_table("users",
    fields = [
        uuid("id", primary_key=True, default="new_uuid"),
        varchar("email", 255),
        varchar("name", 100, nullable=True),
        timestamp("created_at", auto_create=True),
    ],
    indexes = [
        index("idx_users_email", ["email"], unique=True),
    ],
)

Generated SQL (PostgreSQL):

CREATE TABLE users (
    id         UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email      VARCHAR(255) NOT NULL,
    name       VARCHAR(100),
    created_at TIMESTAMP NOT NULL DEFAULT NOW()
);
CREATE UNIQUE INDEX idx_users_email ON users (email);

Down: DROP TABLE users

Parameter Position Type Required Description
name 1st string Yes Table name
fields keyword list of field dicts No Column definitions
indexes keyword list of index dicts No Index definitions
schema_only keyword bool No Track in schema without executing DDL (default: False)

drop_table

Drops an existing table.

drop_table("legacy_sessions")
drop_table("legacy_cache", ignore_errors=True)

Generated SQL: DROP TABLE legacy_sessions

Down: Reconstructs CREATE TABLE from the pre-drop schema state. Because Down needs the original table definition, the schema state at the point of rollback must include it — which it will, as long as you roll back in order.

Destructive. This operation causes data loss.

Parameter Position Type Required Description
name 1st string Yes Table name to drop
schema_only keyword bool No Update schema state without executing SQL (default: False)
ignore_errors keyword bool No Log a warning and continue if the SQL fails (default: False)

rename_table

Renames a table.

rename_table("accounts", "users")

Generated SQL (PostgreSQL): ALTER TABLE accounts RENAME TO users

Down: ALTER TABLE users RENAME TO accounts

Parameter Position Type Required Description
old_name 1st string Yes Current table name
new_name 2nd string Yes New table name

add_field

Adds a column to an existing table.

add_field("users", varchar("phone", 20, nullable=True))

Generated SQL: ALTER TABLE users ADD COLUMN phone VARCHAR(20)

Down: ALTER TABLE users DROP COLUMN phone

Parameter Position Type Required Description
table 1st string Yes Table to modify
field 2nd field dict Yes Field definition from any field builtin
schema_only keyword bool No Schema-only (default: False)

Tip: New columns added to existing tables should usually be nullable=True or have a default value, otherwise the ALTER TABLE will fail if existing rows exist.


drop_field

Removes a column from an existing table.

drop_field("users", "legacy_token")
drop_field("item", "legacy_code", ignore_errors=True)

Generated SQL: ALTER TABLE users DROP COLUMN legacy_token

Down: Reconstructs ADD COLUMN from the pre-drop schema state.

Destructive. This operation causes data loss.

Parameter Position Type Required Description
table 1st string Yes Table to modify
field_name 2nd string Yes Column name to drop
schema_only keyword bool No Schema-only (default: False)
ignore_errors keyword bool No Log a warning and continue if the SQL fails (default: False)

alter_field

Changes the definition of an existing column. Requires both old and new field definitions so the migration can be reversed.

alter_field("users",
    old_field = varchar("bio", 255, nullable=True),
    new_field = text("bio", nullable=True),
)

Generated SQL (PostgreSQL): ALTER TABLE users ALTER COLUMN bio TYPE TEXT

Down: Applies old_field definition back to the new_field column.

Both old_field and new_field must have the same name. The auto-generator fills these in from the schema diff — when writing by hand, provide the full field definition for both.

Parameter Position Type Required Description
table 1st string Yes Table to modify
old_field keyword field dict Yes Current column definition (before change)
new_field keyword field dict Yes Desired column definition (after change)
strategy keyword string No How to handle the type change (see below)

Strategy options:

Strategy Behaviour Data preserved?
"cast" (default) Direct ALTER COLUMN ... TYPE Yes (if the database can cast between types)
"drop_create" Drops the column, then adds it with the new definition No — column data is lost

Use strategy="drop_create" when the database cannot cast between the old and new types (e.g. text → bytes):

alter_field("sessions",
    old_field = text("data"),
    new_field = bytes("data"),
    strategy = "drop_create",
)

rename_field

Renames a column.

rename_field("users", "fullname", "display_name")

Generated SQL (PostgreSQL): ALTER TABLE users RENAME COLUMN fullname TO display_name

Down: ALTER TABLE users RENAME COLUMN display_name TO fullname

Parameter Position Type Required Description
table 1st string Yes Table to modify
old_name 2nd string Yes Current column name
new_name 3rd string Yes New column name

add_index

Adds an index to an existing table.

add_index("orders",
    index("idx_orders_user_created", ["user_id", "created_at"]),
)

Generated SQL: CREATE INDEX idx_orders_user_created ON orders (user_id, created_at)

Down: DROP INDEX idx_orders_user_created

Parameter Position Type Required Description
table 1st string Yes Table to index
index 2nd index dict Yes Index definition from index()

drop_index

Removes an index.

drop_index("orders", "idx_orders_user_created")
drop_index("job", "job_legacy_idx", ignore_errors=True)

Generated SQL: DROP INDEX idx_orders_user_created

Down: Reconstructs CREATE INDEX from the pre-drop schema state.

Parameter Position Type Required Description
table 1st string Yes Table the index belongs to
index_name 2nd string Yes Index name to drop
ignore_errors keyword bool No Log a warning and continue if the SQL fails (default: False)

add_foreign_key

Adds a foreign key constraint to an existing column. The FK column must already exist (created by add_field or create_table). Use this when the FK constraint needs to be added after table creation (e.g., circular references) or when adding a constraint to an existing table.

add_foreign_key("orders", "user_id", "fk_orders_user_id", "users",
    on_delete = "CASCADE",
)

Generated SQL: ALTER TABLE orders ADD CONSTRAINT fk_orders_user_id FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE

Down: ALTER TABLE orders DROP CONSTRAINT fk_orders_user_id

Parameter Position Type Required Description
table 1st string Yes Table to add the constraint to
field_name 2nd string Yes Column that holds the foreign key
constraint_name 3rd string Yes Name of the constraint
referenced_table 4th string Yes Table referenced by the FK
on_delete keyword string No CASCADE, RESTRICT, SET_NULL, PROTECT, etc. (default: "CASCADE")
on_update keyword string No Optional ON UPDATE action (default: "")
ignore_errors keyword bool No Suppress errors (default: False)

drop_foreign_key

Drops a foreign key constraint from an existing table.

drop_foreign_key("orders", "fk_orders_user_id")
drop_foreign_key("job", "fk_job_category_id", ignore_errors=True)

Generated SQL: ALTER TABLE orders DROP CONSTRAINT fk_orders_user_id

Down: Reconstructs ADD CONSTRAINT from the pre-drop schema state.

Parameter Position Type Required Description
table 1st string Yes Table to drop the constraint from
constraint_name 2nd string Yes Constraint name to drop
ignore_errors keyword bool No Log a warning and continue if the SQL fails (default: False)

run_sql

Executes raw SQL directly. This is the escape hatch for anything the typed operations cannot express.

run_sql(
    forward  = "UPDATE users SET status = 'active' WHERE status IS NULL",
    backward = "UPDATE users SET status = NULL WHERE status = 'active'",
)

Up: Executes forward verbatim.

Down: Executes backward verbatim. If rollback is not possible or not needed, omit backward or leave it as an empty string.

State: run_sql does not update the in-memory schema state. If your SQL creates or alters objects that later operations depend on, use typed operations for those.

For multi-line SQL, use triple-quoted strings:

run_sql(
    forward = """
        CREATE OR REPLACE FUNCTION update_modified_date()
        RETURNS TRIGGER AS $$
        BEGIN
            NEW.modified_date = CURRENT_TIMESTAMP;
            RETURN NEW;
        END;
        $$ LANGUAGE plpgsql
    """,
    backward = "DROP FUNCTION IF EXISTS update_modified_date()",
)
Parameter Position Type Required Description
forward keyword string Yes SQL to execute on up
backward keyword string No SQL to execute on down. Empty string means no-op on rollback.

upsert_data

Inserts or updates rows in a table. Designed for seeding reference data (country codes, status enums, configuration rows) as part of a migration. Generates database-appropriate upsert SQL automatically.

Rollback (Down) deletes each row by matching on conflict_keys.

upsert_data("countries",
    conflict_keys = ["code"],
    rows = [
        row(code="AU", name="Australia"),
        row(code="US", name="United States"),
        row(code="NZ", name="New Zealand"),
    ],
)
Parameter Position Type Required Description
table 1st string Yes Target table name
conflict_keys keyword list of strings Yes Columns used to detect conflicts (primary key or unique constraint)
rows keyword list of row dicts Yes Rows to upsert, each built with row()

The row() helper takes keyword-only arguments — each keyword becomes a column name:

row(id="abc-123", name="Widget", price=19.99, description=None)

Values can be strings, integers, floats, booleans, or None (maps to SQL NULL).

Generated SQL by provider:

Provider Dialect
PostgreSQL, AuroraDSQL INSERT … ON CONFLICT (key) DO UPDATE SET col = EXCLUDED.col
MySQL, TiDB, StarRocks INSERT … ON DUPLICATE KEY UPDATE col = VALUES(col)
SQLite, Turso INSERT … ON CONFLICT(key) DO UPDATE SET col = excluded.col
SQL Server, Vertica MERGE INTO … USING … WHEN MATCHED … WHEN NOT MATCHED …
Redshift DELETE … WHERE key IN (…); INSERT …
ClickHouse INSERT … (dedup via ReplacingMergeTree engine)
YDB UPSERT INTO …

upsert_data does not update the in-memory schema state. It is a data operation only.


set_defaults

Registers symbolic default names that field default= values resolve to at runtime. Must appear before any operation that uses the symbolic defaults.

set_defaults({
    "new_uuid": "gen_random_uuid()",
    "now":      "CURRENT_TIMESTAMP",
    "today":    "CURRENT_DATE",
    "zero":     "0",
    "false":    "false",
    "true":     "true",
    "blank":    "''",
    "null":     "null",
    "object":   "'{}'::jsonb",
    "array":    "'[]'::jsonb",
})

When a field specifies default="new_uuid", the runtime resolves it to gen_random_uuid(). This allows the same migration to work across different database engines by changing only the defaults map.

Parameter Position Type Required Description
mapping 1st dict Yes Map of symbolic name to SQL expression

set_type_mappings

Overrides SQL type names for the current database provider.

set_type_mappings({"float": "DOUBLE PRECISION"})
Parameter Position Type Required Description
mapping 1st dict Yes Map of abstract type name to provider-specific SQL type

Writing Custom Migrations

Modifying a generated migration

After morphic generate creates a migration, you can edit it freely before it is applied. Common edits:

Add a data backfill after a column:

operations = [
    add_field("users", varchar("display_name", 100, nullable=True)),
    # Backfill display_name from existing name column
    run_sql(
        forward = "UPDATE users SET display_name = name WHERE display_name IS NULL",
    ),
    # Now make it NOT NULL
    alter_field("users",
        old_field = varchar("display_name", 100, nullable=True),
        new_field = varchar("display_name", 100),
    ),
]

Add a NOT NULL column to an existing table (safe pattern):

operations = [
    # Step 1: add nullable
    add_field("orders", varchar("region", 50, nullable=True)),
    # Step 2: populate existing rows
    run_sql(
        forward = "UPDATE orders SET region = 'unknown' WHERE region IS NULL",
    ),
    # Step 3: tighten to NOT NULL
    alter_field("orders",
        old_field = varchar("region", 50, nullable=True),
        new_field = varchar("region", 50),
    ),
]

Drop a table/column/index that might not exist:

When cleaning up schema drift or writing migrations against databases in an unknown state, set ignore_errors=True so the runner logs a warning and continues instead of aborting:

operations = [
    drop_table("legacy_cache", ignore_errors=True),
    drop_index("users", "idx_temp_sort", ignore_errors=True),
    drop_foreign_key("orders", "fk_orders_old_user_id", ignore_errors=True),
]

ignore_errors=True is available on drop_table, drop_field, drop_index, and drop_foreign_key.

Writing a migration entirely by hand

Create the file manually in the migrations/ directory following the naming convention:

migrations/0005_seed_permissions.star
migration(
    name = "0005_seed_permissions",
    dependencies = ["0004_add_permissions_table"],
    operations = [
        run_sql(
            forward = """
INSERT INTO permissions (name, description) VALUES
    ('users.read',   'Read user records'),
    ('users.write',  'Create and update user records'),
    ('users.delete', 'Delete user records')
""",
            backward = """
DELETE FROM permissions
WHERE name IN ('users.read', 'users.write', 'users.delete')
""",
        ),
    ],
)

Data migration pattern

Use run_sql for data migrations. Keep schema operations (DDL) in separate migrations from data operations (DML) where possible — this makes rollback cleaner.

# 0006_migrate_user_roles.star
migration(
    name = "0006_migrate_user_roles",
    dependencies = ["0005_add_roles_table"],
    operations = [
        run_sql(
            forward = """
INSERT INTO user_roles (user_id, role_id)
SELECT u.id, r.id
FROM users u
JOIN roles r ON r.name = u.legacy_role
WHERE u.legacy_role IS NOT NULL
""",
            backward = """
DELETE FROM user_roles
WHERE (user_id, role_id) IN (
    SELECT u.id, r.id
    FROM users u
    JOIN roles r ON r.name = u.legacy_role
    WHERE u.legacy_role IS NOT NULL
)
""",
        ),
    ],
)

Multiple operations per migration

A migration can contain any number of operations. They run in order on up and in reverse on down:

operations = [
    create_table("posts",
        fields = [
            uuid("id", primary_key=True, default="new_uuid"),
            foreign_key("author_id", fk("users", on_delete="CASCADE")),
            varchar("title", 500),
            text("body", nullable=True),
            boolean("published", default="false"),
            timestamp("created_at", auto_create=True),
            timestamp("updated_at", auto_update=True),
        ],
    ),
    add_index("posts", index("idx_posts_author", ["author_id"])),
    add_index("posts", index("idx_posts_published", ["published", "created_at"])),
]

Merge Migrations

When two developers create migrations independently from the same parent, the DAG develops branches:

0001_initial
├── 0002_add_phone    (developer A)
└── 0003_add_avatar   (developer B)

morphic generate --merge generates a merge migration automatically:

# 0004_merge_0002_add_phone_and_0003_add_avatar.star
migration(
    name = "0004_merge_0002_add_phone_and_0003_add_avatar",
    dependencies = ["0002_add_phone", "0003_add_avatar"],
    operations = [],
)

A merge migration:

  • Has empty operations — it does nothing to the database
  • Has two or more dependencies — pointing at each branch leaf
  • Exists solely to create a common ancestor in the DAG

After a merge migration, subsequent migrations have a single node to depend on again.


Squash Migrations

Squash migrations replace a sequence of old migrations with a single equivalent migration. This keeps the migrations/ directory manageable as a project grows.

morphic generate --squash produces a file like:

migration(
    name = "0001_squashed_0005",
    dependencies = [],
    replaces = ["0001_initial", "0002_add_phone", "0003_add_avatar", "0004_merge", "0005_add_roles"],
    operations = [
        # All operations from 0001–0005, consolidated
        create_table("users", fields=[...]),
        add_field("users", varchar("phone", 20, nullable=True)),
        # ...
    ],
)

replaces tells the runner that if all the named migrations are already applied in the history table, the squash migration itself should be considered applied too (no re-run needed). This allows gradual cut-over without downtime.

For new databases, only the squash migration runs. For existing databases with the original migrations already applied, the squash is automatically treated as applied.


Previewing SQL Before Applying

Before running up, preview exactly what SQL will execute:

morphic migrate showsql

Output:

-- 0005_seed_permissions
INSERT INTO permissions (name, description) VALUES
    ('users.read',   'Read user records'),
    ...

-- 0006_migrate_user_roles
INSERT INTO user_roles (user_id, role_id)
SELECT u.id, r.id FROM users u ...

This is useful for auditing, security review, and debugging provider-specific SQL generation. No database connection is needed — SQL is generated in-process.


File Naming Convention

Generated files follow the pattern:

NNNN_description.star
  • NNNN — zero-padded 4-digit sequence number (0001, 0002, ..., 9999)
  • description — lowercase, words joined by underscores

The sequence number determines registration order when two migrations have the same parents. It does not imply a strict linear order — the DAG is determined solely by dependencies, not file names.

Examples:

0001_initial.star
0002_add_phone.star
0003_add_avatar.star
0004_merge_0002_add_phone_and_0003_add_avatar.star
0005_add_roles.star
0006_seed_permissions.star

Common Patterns Reference

Timestamps

timestamp("created_at", auto_create=True),
timestamp("updated_at", auto_update=True),

Soft delete

timestamp("deleted_at", nullable=True),

UUID primary key

uuid("id", primary_key=True, default="new_uuid"),

Foreign key with cascade

foreign_key("user_id", fk("users", on_delete="CASCADE")),

Decimal money field

decimal("amount", 10, 2),

Unique composite index

index("idx_memberships_user_org", ["user_id", "org_id"], unique=True),

Conditional run_sql (no-op down)

run_sql(
    forward  = "UPDATE config SET value = 'v2' WHERE key = 'api_version'",
    backward = "",  # intentionally irreversible
),

See Also