Fine-tuning Qwen3.5-4B on ADP-v2 (LoRA)

End-to-end pipeline to LoRA fine-tune Qwen/Qwen3.5-4B on four data mixtures drawn from neulab/adp-v2, using LLaMA-Factory — the same SFT pipeline used in the ADP paper (arXiv:2510.24702).

The four subsets (one training run each)

Run	Config	Data mixture
1	configs/subset1_toucan.yaml	toucan_1_5m (enterprise-filtered)
2	configs/subset2_toucan_toolmind.yaml	toucan + toolmind
3	configs/subset3_toucan_dolci.yaml	toucan + dolci_instruct_sft_tool_use
4	configs/subset4_toucan_toolmind_dolci.yaml	toucan + toolmind + dolci

toucan_1_5m has ~1.76M rows. Per your instruction it is filtered to keep enterprise-relevant trajectories (database ops, data analysis, AI/ML, financial services, time/calendar) and capped at ~150k — see filters/enterprise_keywords.py. toolmind (~163k) and dolci (~227k) are used in full.

All three files are built once and composed per-subset via the dataset: field, so toucan is filtered only a single time.

---

Data format (what the raw data is, and what we convert it to)

The ADP-v2 24k shards are stored as OpenAI chat-completions records, not the plain ShareGPT shown in the paper. Each row has:

• messages: [{role, content, tool_calls?, tool_call_id?, name?}] with roles system / user / assistant / tool; assistant tool calls live in structured tool_calls, tool results are role: "tool" messages.
• tools: a list of ~48 OpenAI-style function definitions available to that task.

They live at <config>/24k/full_sft/*.jsonl. datasets.load_dataset cannot load them (the config mixes std/24k/110k schemas → CastError), so 01_prepare_data.py reads the shards directly via huggingface_hub.

01_prepare_data.py renders each trajectory to LLaMA-Factory ShareGPT form (the paper's OpenHands plain-text style), which is robust to the messy real data (multi-turn, interrupted, content+tool-call interleaving):

• the available tools are appended to the system prompt as text, so the model sees what it can call;
• assistant tool calls → <tool_call>{json}</tool_call> inside gpt turns;
• tool results → <tool_response>...</tool_response> human turns;
• consecutive same-side turns are merged so alternation is always valid.

{
  "id": "...",
  "system": "You are OpenHands agent...\n\n# Available tools\n## <name> ...",
  "conversations": [
    {"from": "human", "value": "..."},
    {"from": "gpt",   "value": "I'll ...\n<tool_call>\n{\"name\": ..., \"arguments\": {...}}\n</tool_call>"},
    {"from": "human", "value": "<tool_response>\n...\n</tool_response>"},
    {"from": "gpt",   "value": "..."}
  ]
}

02_make_dataset_info.py registers this as sharegpt (role_tag=from, content_tag=value, user_tag=human, assistant_tag=gpt, system column).

Why not LLaMA-Factory's native openai converter? It requires strict user/assistant alternation and drops ~38% of these trajectories (user follow-ups after a tool result) plus the assistant's narration. The plain-text render keeps essentially all of it.

Toolchain notes (this environment)

• flash-attn couldn't compile (system nvcc 12.0 vs torch cu13) → configs use flash_attn: sdpa.
• transformers 5.14 (needed for Qwen3.5) is newer than LLaMA-Factory 0.9.6's pin (<=5.8) → training/merge scripts set DISABLE_VERSION_CHECK=1.
• neat_packing builds a 4D block-diagonal mask that needs flash-attn; with sdpa it crashes (attention-mask size mismatch) → set neat_packing: false (plain packing still gives the throughput win). Verified via smoke test.
• Qwen3.5 is a hybrid model (GatedDeltaNet linear-attention layers). Those need flash-linear-attention + tilelang, and tilelang JIT-compiles Hopper kernels with nvcc from CUDA 13 (CUDA_HOME=/usr/local/cuda-13.0). Without this the linear-attention layers fall back to a ~150x slower torch path (485 s/step). 03_train.sh exports CUDA_HOME for this reason. (causal-conv1d won't build here but isn't required; the conv falls back with minor cost.) Data must also be sanitized of literal <image>/<video>/<audio> tokens (done in 01_prepare_data.py) since Qwen3.5's template is multimodal.
• Throughput reality (measured, ~7k tok/s per H200 at 24k): the enterprise-filtered toucan trajectories average ~23k tokens, so 150k of them = ~3.45B tokens/epoch. Full config (150k x 3 epochs x 24k) ≈ 2+ weeks per subset. Reduce scope (fewer toucan examples / fewer epochs / lower cutoff) for feasible runs. A 500-step trial is ~11.5 h/run at ~83 s/step.

---

Confirmed hyperparameters (LoRA SFT)

Group	Setting	Value	Notes
Model	base	Qwen/Qwen3.5-4B	released Mar 2026
	template	qwen3_5_nothink	non-thinking; matches direct-action data
	precision	bf16
LoRA	finetuning_type	lora
	lora_target	all	all linear layers
	lora_rank	16	bump to 32 for more capacity
	lora_alpha	32	= 2 × rank
	lora_dropout	0.05
Optim	learning_rate	1.0e-4	typical LoRA LR (10× full-FT)
	scheduler	cosine
	warmup_ratio	0.03
	weight_decay	0.0
	optimizer	adamw_torch	LLaMA-Factory default
Batch	per_device_train_batch_size	1	with 24k packed seqs
	gradient_accumulation_steps	16	effective batch = 16 × #GPUs
Length	cutoff_len	24576	matches the 24k split
	packing	true	throughput win
	neat_packing	false	needs flash-attn; off for sdpa
Schedule	num_train_epochs	3.0	see runtime note below
Misc	gradient_checkpointing	true
	flash_attn	fa2	remove if flash-attn not installed
	val_size	0.01	small held-out set for eval loss

Rationale. The ADP paper full-fine-tuned Qwen-*-Instruct with LLaMA-Factory. You asked for LoRA on 1–2 shared GPUs, so this uses the standard LoRA recipe (rank 16 / α 32, LR 1e-4) with 24k-token packing to keep throughput high. All four runs share these settings for a clean, comparable ablation — the only thing that changes across runs is the data mixture.

Runtime knob — num_train_epochs. Set to 3 epochs (paper-typical). Subset 4 is ~540k trajectories at 24k tokens each, so on 1–2 GPUs this is many GPU-hours — plan accordingly, or drop to 1.0–2.0 in configs/*.yaml for a faster first pass. (Packing keeps throughput high, so wall-clock is far below naive #examples × seq_len.)

⚠️ Adjust before running if your hardware differs. These target 1–2×H200 (143 GB). On smaller GPUs, lower cutoff_len (e.g. 8192) and/or add quantization_bit: 4 (QLoRA).

---

⚠️ Qwen3.5 compatibility caveat (read before Step 3)

Qwen3.5-4B is very recent (Mar 2026) and post-dates most pinned tool versions. Two things to verify at setup:

1. transformers can load the architecture — hence we install from the newest releases in 00_setup_env.sh; you may need transformers from source.
2. LLaMA-Factory has a matching chat template — the configs use qwen3. If Qwen3.5 needs a different/newer template (it is a hybrid thinking model, and may be vision-capable), Step 3 tells you how to check and what to change.

Automatic fallback (built in). You don't have to switch models by hand. scripts/03_train.sh, 04_merge_lora.sh, and 05_inference_test.py all call scripts/_resolve_model.py, which tries Qwen3.5-4B and, if the toolchain can't load it, automatically falls back to Qwen/Qwen3-4B-Instruct-2507 — printing a loud, unmissable banner so you know the fallback happened:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!  MODEL FALLBACK ACTIVATED
!!!  Qwen3.5-4B could NOT be used on this toolchain.
!!!  Reason : transformers cannot load Qwen/Qwen3.5-4B (...)
!!!  Now using FALLBACK base model : Qwen/Qwen3-4B-Instruct-2507
!!!  >>> This run's training / merge / inference all use Qwen3-4B, NOT Qwen3.5. <<<
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

The resolved model + template are passed to LLaMA-Factory as CLI overrides, so training, merging, and inference all stay consistent on whichever model won. To change the fallback target, edit FALLBACK in scripts/_resolve_model.py.

---

Step-by-step

All commands run from /home/ssampat/adp_ft inside the adpft venv.

Step 0 — (optional) Hugging Face auth

neulab/adp-v2 is public, but if you hit a gated/rate-limit error:

source /home/ssampat/adpft/bin/activate
huggingface-cli login          # or: export HF_TOKEN=hf_xxx

Step 1 — Install the training stack (once)

bash scripts/00_setup_env.sh

Installs torch, LLaMA-Factory (from source), the HF stack, and flash-attn into /home/ssampat/adpft. Ends with a sanity check printing torch/GPU/transformers versions.

Step 2 — Build the datasets

source /home/ssampat/adpft/bin/activate
python scripts/01_prepare_data.py --datasets toucan toolmind dolci --target-toucan 150000
python scripts/02_make_dataset_info.py

• Streams toucan (no full download), scores each row for enterprise relevance, keeps the top ~150k -> data/toucan_enterprise.jsonl.
• Downloads toolmind + dolci in full.
• Writes data/dataset_info.json for LLaMA-Factory.

Tune the filter: edit keyword lists in filters/enterprise_keywords.py, or pass --min-score, --target-toucan, then re-run with --overwrite.

Step 3 — Check which model will be used (automatic fallback)

source /home/ssampat/adpft/bin/activate
python scripts/_resolve_model.py

This tries Qwen3.5-4B and prints either Model check OK: using PRIMARY ... or the loud fallback banner shown above (dropping to Qwen/Qwen3-4B-Instruct-2507). The training/merge/inference scripts call this automatically — you don't need to edit any config by hand. If you want Qwen3.5 to work and it's falling back because transformers can't load it, update transformers: pip install -U git+https://github.com/huggingface/transformers.

Optional — dry-run the data pipeline (tokenizes a few samples, no training):

llamafactory-cli train configs/subset1_toucan.yaml max_steps=1

Step 4 — Train (one run per subset)

Pick the GPU(s) you own on the shared box. Currently GPU 3 is the free one:

# single GPU (GPU 3)
CUDA_VISIBLE_DEVICES=3   bash scripts/03_train.sh configs/subset1_toucan.yaml
# two GPUs (DDP), if/when more free up
CUDA_VISIBLE_DEVICES=3,0 bash scripts/03_train.sh configs/subset4_toucan_toolmind_dolci.yaml

Run the four configs to produce the four models. Adapters land in output/<run_name>/; logs in output/logs/; metrics in output/<run_name>/trainer_log.jsonl.

Run one subset at a time (or on separate GPUs) — don't oversubscribe shared GPUs.

Live metrics & plots (built in). Train loss is logged every 10 steps; eval loss + token accuracy every 500 steps (val_size: 0.01). 03_train.sh auto-launches scripts/06_monitor.py in the background, which refreshes these plots every 2 minutes and writes a final snapshot at the end — no second terminal needed:

• output/<run_name>/plots/loss.png — train + eval loss vs step
• output/<run_name>/plots/eval_accuracy.png — eval token accuracy vs step
• output/<run_name>/plots/learning_rate.png — LR schedule

Healthy run: train loss falls then flattens, eval loss tracks it without rising (rising eval loss = overfitting → lower epochs/LoRA rank), eval accuracy climbs. To watch a run from another terminal, or after the fact:

python scripts/06_monitor.py --run subset1_toucan            # live, every 120s
python scripts/06_monitor.py --run subset1_toucan --once     # one snapshot

Step 5 — Smoke-test an adapter

python scripts/05_inference_test.py --run subset1_toucan

Step 6 — (optional) Merge LoRA for deployment

bash scripts/04_merge_lora.sh subset1_toucan
# -> output/subset1_toucan_merged/  (standalone, serve with vLLM/transformers)

---

Project layout

adp_ft/
├── README.md                      # this guide
├── configs/                       # 4 LoRA training configs (one per subset)
├── filters/enterprise_keywords.py # editable enterprise-relevance scorer
├── scripts/
│   ├── 00_setup_env.sh            # install stack into adpft venv
│   ├── 01_prepare_data.py         # download + filter -> ShareGPT jsonl
│   ├── 02_make_dataset_info.py    # register datasets with LLaMA-Factory
│   ├── _resolve_model.py          # Qwen3.5 -> Qwen3-4B fallback w/ loud banner
│   ├── 03_train.sh                # launch a training run
│   ├── 04_merge_lora.sh           # merge adapter into base weights
│   ├── 05_inference_test.py       # quick generation smoke test
│   └── 06_monitor.py              # live loss/accuracy plots during training
├── data/                          # generated jsonl + dataset_info.json
└── output/                        # adapters, logs, plots