Add VLM export support (SmolVLM as starting point)

[ananyajoshi0203-cpu]

Hey, I recently opened a PR in huggingface/transformers (#46360) adding vision/audio input support to the ExecuTorch export pipeline in executorch.py. The reviewer (zucchini-nlp) closed it and pointed me here, mentioning that multimodal ExecuTorch support is moving to optimum-executorch.

Looking at the repo, I can see text LLMs and image classifiers are both supported, but there's no export path for vision-language models, models that take image + text and generate text. SmolVLM seems like an obvious first target: it's a HF model, it's small enough to actually run on-device.

Before I start on anything, a few things I'd want to confirm:

Is VLM export support already being worked on somewhere, or is it an open gap?
Would the right shape be a new ExecuTorchModelForImageTextToText class following the ExecuTorchModelForCausalLM pattern?
Any known blockers on the ExecuTorch side for exporting a vision encoder + LLM decoder in one graph?

Happy to prototype something if this direction makes sense, just don't want to duplicate work.

[IlyasMoutawwakil]

Hi ! There's currently an effort to support exporting to dynamo/onnx/executorch in Transformers directly (https://github.com/huggingface/transformers/pull/41992/changes) please try it out and tell us what you think. It should be possible to export a vlm end-to-end in one graph with the help of (huggingface/transformers#45396). But it's still impractical from an inference standpoint. We decompose the vlm in HF Exporters because imo it's still the most natural way from an inference standpoint.

[ananyajoshi0203-cpu]

Thanks for the context! I'll take a look at the Transformers export effort and try it out. The decomposed approach makes sense for inference and I'm happy to contribute there if there's an open gap for VLM support in optimum-executorch. Is SmolVLM a good first target, or is there a model you'd prioritize?