Add image_features_indices: declared base64 image columns become DINOv3 features

[alanprior]

What & why

People keep asking for TabPFN on images plus tabular fields, medical imaging in particular. This adds the recipe of the MulTaBench paper (https://arxiv.org/abs/2605.10616), where TabPFN on frozen DINOv3 embeddings beats the alternatives on image-tabular tasks: an image sits in a DataFrame cell as a base64 string (or bytes), the user names the column in image_features_indices, and TabPFN swaps the column for a small block of numeric features before validation. Nothing is detected.

How it works

• preprocessing/images.py::ImageTransformer mirrors TextTransformer. base.expand_images runs first, then expand_dates and expand_text, each returning the expanded input, its fitted transformer and the declared categorical positions in the expanded input. Each declared image column is decoded, embedded by a frozen DINOv3 ViT-S/16 (CLS token, 384 dims, batches of 64, no grad), standardised and reduced by a PCA fit on the training rows, to min(IMAGE_N_COMPONENTS, n_rows, 384) features named <column>_img_<i>.
• The encoder lives in a module-level cache keyed by model id and is never stored on the transformer, so save_fitted_tabpfn_model files stay small; estimator_to_device re-points the transformer's device on load.
• A missing or undecodable cell is refused by column and row, at fit and at predict; a zero row would mean "the mean training image". A changed embedding width at predict is refused, as is an array after an expanding fit.
• Estimators expose image_transformer_; categorical_features_indices_ now gives the declared positions after image, date and text expansion; tuning clones fit on the already-embedded array. Differentiable input rejects declared image columns. The fine-tuning estimators do not run it.

How to use

pip install "tabpfn[image]"   # transformers, torchvision, pillow

TabPFNClassifier(image_features_indices=[3])

The default encoder's weights are gated: accept the license at https://huggingface.co/facebook/dinov3-vits16-pretrain-lvd1689m and run hf auth login once, or point inference_config={"IMAGE_ENCODER_MODEL": ...} at an ungated encoder.

examples/tabpfn_with_images.py runs three MulTaBench tasks (Amazon Bestseller by default, Glaucoma SMDG and Hateful Meme selectable) with tabular-only, picture-only and combined models. The default run takes about two minutes on an A100 and gives R2 0.164 / 0.569 / 0.633 for the three.

Design decisions

• Declared only, never sniffed: an undeclared base64 column is left alone and none of the optional dependencies is imported.
• TRANSFORM_IMAGE defaults to True since it only acts on declared columns; off, declared columns are refused with a message naming the flag, like dates.
• Optional extra rather than a hard dependency; torchvision is in it because transformers ships DINOv3 with a fast image processor only.
• No L2 normalisation and no fine-tuning: the paper's frozen protocol.
• Tests run against a stub encoder so CI downloads nothing; one slow test uses the real encoder and skips without the extra or a Hub token.

Deferred follow-ups

• File paths or URLs as cells (only base64 and bytes for now).
• Larger encoders or a fine-tuned head.

🤖 Generated with Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 795573a. Configure here.}

[cursor]

Changelog fragment exceeds user-facing scope

Low Severity

The towncrier fragment is a long implementation write-up rather than a one- or two-sentence user note. It names fitted internals such as image_transformer_ and categorical_features_indices_, and retells encoder, PCA, refusal, extra, and fine-tuning details that belong in the PR body, not the concatenated user changelog.

 

^{Triggered by learned rule: Towncrier fragments: user-facing change only; keep concise}

^{Reviewed by Cursor Bugbot for commit 795573a. Configure here.}

[alanprior]

We will reimplement this in TabPFN-Extensions