[API Request] Advanced transcoding

[Tolriq]

Type of change

API extension

Proposal description

This is the first step to discuss the future transcoding API extension that give a lot more control over when and to what to transcode.

This is necessary to give a lot more liberty to clients to request media they can play or cast to other device with proper quality control. When you can't play DSD files for example, it's better to transcode to FLAC than low quality mp3.
Or when your device does not support sampleRate over 48Khz it's better to reduce sample rate then low quality mp3.

There is 2 different approach to this need that have different impacts on the client or the servers.

1 - Client side decision

Leave the transcoding decision and target to the client

• The client request the precise details about the media with codec, container and sub codec details
• The client decide what to do
• The client request transcoding to a target configuration (codec, container, max sample rate, max bitrate, ...)

Issues:

• Not all servers may support all the possible target configuration depending on their ffmpeg version or build, it requires error handling and fallback client side.
• Each clients needs to write their transcoding decision engine.

2 - Server side decision

Leave the transcoding decision and target to the server.

• The client send a request to the server telling I want to play that media, I support X,Y,Z profile for direct play, if you transcode I prefer A,B,C profile (in order)
• The server answer: You can direct play or you can transcode and pass an object that will need to be passed to the transcode endpoint, or I'm unable to satisfy your request.
• The client plays or cast the media.

Issues:

• Requires a precise API definition for the profiles (Need to decide the level of control to avoid changing the API too often)
• Each server needs to write the transcoding decision engine.

Other platforms

Other platforms like Emby, Plex, Jellyfin all do solution 2. But since they are a single server for a give API endpoint there's no drawback of multiple servers writing their own transcoding decision engine.

Backward compatibility impact

No response

Backward compatibility

• No backward compatibility impact.

API details

TBD

Security impacts

No response

Potential issues

No response

Alternative solutions

No response

[epoupon]

Thank you for starting this discussion.
I would go with option 2. I feel like it’s not too complicated to implement on the server side. And ideally, if everything goes well, we should have more OS clients than servers anyway, so it makes sense to place this load on the servers.

[Tolriq]

No it requires a lot more control, codec, container, maxbitrate, max sample rate, max channel number, max bitdepth at minimum .Those are not server side profiles but client profiles that vary depending on the need.
The client can't know in advance what the server support. It request what it support (the client) and the servers says if it can or not.

[sentriz]

doesn't seem backwards compatable with what we have already. why not add an endpoint for the server to respond with what it can support, client chooses from the list, and sends in the next request with format= (we already have this) and maxBitRate= (we also have this one)

for control over num channels and samplerate, what's the usecase there?

[Tolriq]

The use case is the basis of the explanation I gave on why the new endpoints.
So no it's not meant to be backward compatible it's new extension and new endpoints.

For my use case, I have TB of hi res music, the Sonos one does not support FLAC 24/96 or more.
I don't want to transcode that to mp3 or opus, I want FLAC 16/96 or 24/48.
That's specific to that device and can't be preconfigured on the server.

Same for DSD, my phone only support up to 192 so I want FLAC 24/192 and not opus or FLAC 16/44.

Same for channel count there's plenty of usage to request multi channel music to stereo without quality change.

[deluan]

I also vote for option 2, same reason as @epoupon. Don't we have a previous discussion about that already? Should we add the summary of that discussion here, or at least a link to it, for reference?

[Tolriq]

It was mixed in the discussion about the codec and container data for songs that we decided to postpone.

This one being self contained I prefer to move on that first before going back on the global song data where no one agreed on anything.

But yes I also think 2 is way better, option 1 was more if servers did not want to work on the engine decision part.

[simonbun]

Should the application of ReplayGain be part of this change (driven from the client as only it can handle auto)?

This is one of the areas currently lacking in my LMS & Symfonium setup when casting.

[simonbun]

I've been through the app settings a number of times - because I didn't want exactly this situation to occur - and the only place I've seen ReplayGain mentioned is for local playback (Settings > Playback > Playback > Equalizer - DSP) or by proxy (Playback settings > Decoding and transcoding > Casting replay gain processor (Experimental)), which I really want to avoid.

Apologies if I've missed another option.

[Tolriq]

So you want replay gain for casting, but want to avoid the option that do replay gain for casting ?

[simonbun]

I don't want my phone proxying the stream.

I thought this discussion was about server transcoding?

[Tolriq]

It is but you said the feature was not possible when it's in fact possible ;)

[simonbun]

Ok you win, but if I route the audio through the phone then I lose one of the sticks to beat AirPlay users with :)

Hopefully ReplayGain will make it into this extension (or another) as it's the one caveat I have to make when recommending OpenSubsonic as a solution.

Great job everyone!

[benjamindaines]

I would also add to the discussion, the ability to NOT transcode, including to raw. The issue I've been running into is related to DSD playback, especially via UPnP. I use Symfonium as my playback app of choice, as well as a mechanism to stream to UPnP devices.

What I've found to be what seems like an issue with opensubsonic itself is that DSD files get converted to "raw" when streaming them via UPnP, and wrapped in a .flac container. This confuses the heck out of renderers. Out of 3 renders, only one can make sense of this type of stream. Upmpd (MPD with libupnp bolted on, basically) handles this. However, Neutron and UAPP cannot make sense of it, and want to treat the stream as PCM. Neutron does recognize the stream as DPCM (PCM containing DSD), but still cannot manage to play it properly. I've been emailing back and forth with the folks who make Neutron, and thus far, the only solution we've been able to come up with, is to enable PCM to DSD conversion, forcing the Neutron to do the DSD thing. This, however, is far form ideal, as it greatly increases CPU usage (15% when playing DSD from Neutron itself, to 83% or more when doing the conversion work around).

I've isolated (as best I can) this to be a function of opensubsonic itself. The server I use is Navidrome, but the same behavior is exhibited when using lms as the server. Symfonium properly stream .dsf / dsd to UPnP renderers when not using a subsonic media provider. So the constant here is opensubsonic.

I don't believe this would be an issue if either the .dsf file was served up without being converted to "raw," especially when being sent to an external player (this would likely require communication from the client to the server), or some how come up with a solution for DSD that otherwise works.

It seems, to me, that everything was designed with only PCM in mind. Which, is not unreasonable. Probably 96% of music libraries are lossy or PCM. However, DSD is a thing, and it should be supported properly.

I've talked to folks working on Navidrome, Neutron, and Tolriq over at Symfonium. Everyone claims their portion of the equation is working properly (with the exception of Nuetron not properly broadcasting .dsf / dsd capabilities, but fixing that did not result in any different behavior). Thus, it must be a design issue / oversight.

Even the ability to manually specify rules based on file type / codec / bitrate* via server the config file would be fine by me. Clearly, this would be far from the best, most elegant solution, but I'd take it.

Either that, or everyone needs to switch their UPnP renders to using libupnp, as it is the most modern implementation and seems to be the only thing that works properly :)

It's probably a bit unfair for me to editorialize this much, as I lack the technical understanding of how osub works, but these are just my observations as a frustrated user. I'm happy to test anything that needs testing.

• if this is possible via editing Navidrome's transcode rules / "profiles" please let me know! Logically, it seems like that should be possible, but I can't make heads or tails of how to do it.