Back2Struct: Making Structured Images Editable Again

Pengyu Yan  ·  Yixin Wu  ·  Yunjie Tian  ·  David Doermann
University at Buffalo, SUNY

Links: 📄 Paper (arXiv) · 🌐 Project Page · 🤖 Model · 📚 Dataset

Recovering the reference image across the most advanced LLMs — GPT-5 (Thinking), Qwen3-Max (>1T params), Gemini-2.5-Pro — and Back2Struct (ours). With only 7B parameters, Back2Struct reconstructs structured graphics with negligible confusion or mistakes.

Abstract

Structured images — diagrams, charts, and flowcharts — are inherently symbolic and can be compactly represented in an editable format, yet in practice they are shared as raster images and are no longer graphically editable. We present Back2Struct, which "makes structured images editable again" by directly recovering vector graphics code (SVG / XML) from image representations. Given an image of a structured graphic, Back2Struct predicts semantically object-level SVG/XML that explicitly encodes text, shapes, topology, and layout — rather than low-level pixel vectorization — so the code imports straight into tools like PowerPoint to edit, restyle, and reuse while preserving structural fidelity. Beyond supervised fine-tuning on ground-truth SVG token sequences, we optimize Back2Struct with reward-based learning (a composite reward encouraging compilability, length consistency, and structural/semantic similarity), improving accuracy, editability, validity, and user alignment over strong baselines.

Highlights

• StructHub — 84K paired structured raster images and their SVG/XML code, plus a 1,000-sample evaluation benchmark.
• Object-level recovery — vector code that explicitly models text, shapes, topology, and layout (not pixels).
• Reward-based optimization — SFT then GRPO with a composite reward for deployment-time validity and faithfulness.

Results

StructHub benchmark (render-gated). SR = render success (%); GPT = 0–100 GPTScore.

Model	SR ↑	DINO ↑	LPIPS ↓	SSIM ↑	GPT ↑
Overall
Qwen3-VL-30B-A3B	18.8	0.1566	0.9067	0.1201	13.29
StarVector-8B	31.4	0.2694	0.7936	0.2170	21.05
Qwen2.5-VL-7B (base)	52.3	0.3527	0.7834	0.3276	27.56
Qwen2.5-VL-7B (SFT)	42.8	0.3715	0.7403	0.2891	33.49
Back2Struct (RL)	71.5	0.5622	0.6473	0.4966	51.85
Easy
Qwen3-VL-30B-A3B	26.5	0.2231	0.8659	0.1702	19.51
StarVector-8B	46.1	0.3923	0.7023	0.3123	31.90
Qwen2.5-VL-7B (base)	69.3	0.4776	0.7123	0.4196	39.66
Qwen2.5-VL-7B (SFT)	56.0	0.4866	0.6630	0.3789	44.58
Back2Struct (RL)	84.0	0.6384	0.5956	0.5796	62.63
Medium
Qwen3-VL-30B-A3B	19.6	0.1631	0.9004	0.1288	13.38
StarVector-8B	37.0	0.3232	0.7533	0.2583	25.27
Qwen2.5-VL-7B (base)	51.8	0.3546	0.7816	0.3306	26.88
Qwen2.5-VL-7B (SFT)	45.5	0.3985	0.7161	0.3091	36.50
Back2Struct (RL)	75.6	0.6088	0.6182	0.5275	55.14
Hard
Qwen3-VL-30B-A3B	10.2	0.0836	0.9536	0.0614	7.01
StarVector-8B	11.1	0.0932	0.9249	0.0808	6.02
Qwen2.5-VL-7B (base)	35.7	0.2263	0.8561	0.2331	16.17
Qwen2.5-VL-7B (SFT)	27.0	0.2297	0.8413	0.1795	19.43
Back2Struct (RL)	55.0	0.4399	0.7278	0.3830	37.83

On successfully compiled outputs, the 7B model matches or beats frontier closed models:

Model	DINO ↑	LPIPS ↓	SSIM ↑	GPT ↑
Gemini-2.5-Pro	0.8855	0.4648	0.6532	77.95
GPT-5	0.8852	0.4350	0.6573	77.21
Back2Struct (7B)	0.8697	0.3896	0.6738	78.63

Model & Dataset

• 🤖 Model: Pengyu965/Back2Struct-Image2SVG-7B — Qwen2.5-VL-7B fine-tuned (SFT + GRPO) for image→SVG.
• 📚 Dataset: Pengyu965/StructHub — train (86,102) + benchmark (1,000) splits, columns image / svg (+ id, source, token_count).

Run the model

import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model_id = "Pengyu965/Back2Struct-Image2SVG-7B"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [{"role": "user", "content": [
    {"type": "image", "image": "diagram.png"},
    {"type": "text",  "text": "Convert this structured image into editable SVG/XML code."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs,
                   padding=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=8192, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])

Load the dataset

from datasets import load_dataset
ds = load_dataset("Pengyu965/StructHub")
ex = ds["benchmark"][0]
ex["image"].save("input.png"); print(ex["svg"][:500])

Project page

This repository also hosts the project website — see index.html, live at https://pengyu965.github.io/Back2Struct.github.io/. Build/deploy notes are in DEPLOY.md.

Citation

@misc{yan2026back2structmakingstructuredimages,
      title={Back2Struct: Making Structured Images Editable Again},
      author={Pengyu Yan and Yixin Wu and Yunjie Tian and David Doermann},
      year={2026},
      eprint={2609.37016},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.37016},
}