[Inference] fal-ai: audio-to-audio fix + text-to-audio + NeMo ASR & PersonaPlex

[jetjodh]

Summary

Adds and fixes audio model support for the fal-ai inference provider.

1. Fix audio-to-audio for fal-ai audioToAudio() now resolves url/headers/signal via makeRequestOptions and forwards them to getResponse (mirroring imageSegmentation), so the fal queue task can poll for results instead of throwing "URL and headers are required for audio-to-audio task". Input data-URL MIME handling reuses FAL_AI_AUDIO_MIME_MAP (consistent with the ASR fix), so audio/wav/audio/webm/etc. aren't rejected by fal's data-URL decoder.

2. Add text-to-audio support. New queue-based FalAITextToAudioTask (handles both audio_file and audio result shapes), a textToAudio() task function (auto-exposed on InferenceClient via the tasks barrel), a widened TextToAudioTaskHelper.getResponse signature, and export of the (previously defined-but-unexported) text-to-audio inference types from @huggingface/tasks.

3. ASR: handle NeMo/nemotron output shape + timestamps. fal's nemotron ASR endpoint returns the transcript under output (with a partial flag), not text like whisper — so the existing helper would reject it. getResponse now parses both text and output, and normalizes timestamps from chunks (whisper) or segments to HF's AutomaticSpeechRecognitionOutput.chunks.

Testing

• New unit tests: fal-ai audio-to-audio, text-to-audio, and automatic-speech-recognition (text/output/chunks/segments parsing).
• Verified end-to-end against the live fal API: nemotron ASR (correct transcript) and PersonaPlex batch audio-to-audio (audio returned).
• tsc --noEmit, eslint, and the new vitest specs all pass.

---

Note

Medium Risk
Changes the public audioToAudio() request/response path and fal ASR parsing; mistakes could break fal queue audio flows or reject valid ASR payloads, though behavior is covered by new unit tests.

Overview
fal-ai audio-to-audio is wired end-to-end: the task is registered, a queue-based FalAIAudioToAudioTask sends remapped audio_url payloads and polls until it can download returned audio as base64 AudioToAudioOutput[]. audioToAudio() now uses per-provider preparePayloadAsync and passes url/headers/signal into getResponse (same pattern as image segmentation) so queue tasks no longer fail with missing URL/headers.

ASR on fal-ai accepts transcripts in text (Whisper) or output (NeMo/nemotron) and optionally maps chunks or segments into HF AutomaticSpeechRecognitionOutput.chunks. Input encoding for ASR and audio-to-audio shares buildFalAiAudioDataUrl so browser MIME types (e.g. audio/webm) map to labels fal’s data-URL decoder accepts.

AudioToAudioTaskHelper and hf-inference gain preparePayloadAsync; image-to-image fal payload assembly is tightened (parameters + image_url / image_urls without duplicate spreads). New vitest coverage for fal audio-to-audio queue flow and ASR response shapes.

^{Reviewed by Cursor Bugbot for commit 02cd2e2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[hanouticelina]

Thanks @jetjodh! did you test text-to-audio via live testing? Also would it be possible to split the audio-to-audio and text-to-audio support addition into two separate PRs? 🙏

[hanouticelina]

Suggested change

	throw new InferenceClientProviderOutputError(
	`Received malformed response from Fal.ai text-to-audio API: expected { audio_file: { url: string } } result format, got instead: ${JSON.stringify(
	result,
	throw new InferenceClientProviderOutputError(
	`Received malformed response from Fal.ai text-to-audio API: expected { audio_file: { url: string } } or { audio: { url: string } } result format, got instead: ${JSON.stringify(
	result,

[jetjodh]

Thanks @jetjodh! did you test text-to-audio via live testing? Also would it be possible to split the audio-to-audio and text-to-audio support addition into two separate PRs? 🙏

can do that

[jetjodh]

@hanouticelina Done 🙏

• Split: moved text-to-audio to [Inference] Add fal-ai text-to-audio support #2249. This PR now covers audio-to-audio
• Live testing: yes — text-to-audio and ASR were also live-tested.