Why is answer_videos_vlm processor more VRAM hungry than outside a container?

[geoffroy-noel-ddh]

Two implementations of the same model (qwen3-vl-32b-instruct) doing the same tasks on the same input. But they have different VRAM requirements on the same environment.

One in the experiments/qwen folder of the ISSA repository. It works well on a h100 or A100 80g GPUs on e-Research HPC.

The other in the processor of the answer_videos_vlm operator. It goes out of memory (OOM) on the same compute nodes.

Why?

[stephenmcconnachie]

Could it be the batching strategy or other load construction strategy that feeds the data to the inference? I have found that load construction / batching is critical when using our H-100, and it is easy to OOM with same data / same model / same GPU, but with slightly different Python tooling. First thing I would do is check the suyrrounding code / tooling.

[geoffroy-noel-ddh]

Ah, thanks for the tips! The two implementations use the same framework (HF transformers package for python) and should have the same parameters. My guess is on a minor discrepancy in the dependencies (although both use flash attention 2 which considerably reduce the VRAM consumption) or something about the containerisation.

Since you're interested in that issue @stephenmcconnachie I've also tried to use quantised variants of Qwen3-vl-32b-instruct models but without much success yet (its architecture is still quite new and not well supported by all inference frameworks). Ideally I'd like to see a model with that kind of quality running on a 24GB GPU. That would make video question answering with FrameSense much more accessible to consumer hardware.

[stephenmcconnachie]

Ah that's great expereimenting, thank you. I'm so locked in on our H100 that I forget not everyone has 80G to use! Also if it uses lower % of GPU it would mean parallel processes can run with same GPU instead of switching.

[stephenmcconnachie]

I am hoping to install framesense soon and run some processes