Client metadata

[iLLiCiTiT]

PR Checklist

• Add option to define client data using client.json.

Description of changes

Allow to define dependencies, and other possible client metadata with client.json instead of pyproject.toml.

This is a proposal. Right now we do allow to define dependencies using pyproject.toml, and we also do expect that the dependencies are defined with tool.poetry.dependencies. That has multiple issues, we don't use poetry anymore, using pyproject.toml is misleading for developers and for IDEs as they try to define more 'pyproject.toml' metadata that either should not be defined or has no effect.

Example change:
pyproject.toml

[tool.poetry.dependencies]
PyYaml = "^6"

[ayon.runtimeDependencies]
pillow = "^12"

client.json

{
    "dependencies": {
        "PyYaml": "^6"
    },
    "runtimeDependencies": {
        "pillow": "^12"
    }
}

Technical details

The structure of the json is not validated -> thus defined, has to be added to documentation.

We also can't just change it and support both until all addons are changed.

[BigRoy]

I'm not too keen on this. @antirotor I think your thoughts may weight well here.

But client.json tells me very little of the intent? Also, I feel deviating from relying on known structures, like pyproject or similar is just asking for more questions. I also like the fact that tools do know how to author pyproject files if someone wanted to author it that way - the JSON is completely custom and just means more maintenance.

Can't we store it in some dedicated grouped section in a pyproject?

[iLLiCiTiT]

Also, I feel deviating from relying on known structures, like pyproject or similar is just asking for more questions.

For me the issue with pyproject is that it is misleading as it looks like you can use it as standard pyproject.toml, but you can't.

• Whatever you define in pyproject toml except tool.poetry.dependencies and ayon.runtimeDependencies is never used.
• tool.poetry.dependencies are not used for poetry and you should not define things the same way you would define in standard pyproject.toml.
• It is not clear that it is related to client only.
• It requires tomllib, whereas json is python builtin.

We're using something that is known, for something else, because "it is known", but it is not the same. Just because we know that (and how it is used), we're ok-ish with that, but it is completely wrong. Especially for someone who tries to understand. Having pyproject.toml is shortcut leading to wrong assumptions.

Can't we store it in some dedicated grouped section in a pyproject?

What else would be in the pyproject.toml ?

[antirotor]

Can't we store it in some dedicated grouped section in a pyproject?

I would prefer that. You can use dependency groups for that - having the AYON dependencies defined as standard dependencies in main pyproject.toml and adding runtime dependencies via group.

[project]
name = foo
# ...
dependencies = [
  "PyYaml ~= 6"
]

[dependency-groups]
ayonRuntime = [
  "pillow ~= 12"
]

Reasons why this is better

• one standard place do define dependencies
• you need to install these locally for running unit tests and other tasks anyway

Caveats
The only downside I see is that we'll need more careful deps group management

Required changes
Create package script would need to bundle repo pyproject.tom with the zip - making pyproject.toml in repo root mandatory.
Small changes in dependencies-tool - we'll just have to add --group argument probably.

[iLLiCiTiT]

I would prefer that. You can use dependency groups for that - having the AYON dependencies defined as standard dependencies in main pyproject.toml and adding runtime dependencies via group.

We also do expect that the dependencies are as dict key -> package, value -> version which is not how standard pyproject toml does define it. And again it still does not solve the issue that we're using pyproject.toml as something that is known but we don't use it that way, WHICH IS MISLEADING!!!

[antirotor]

Having one pyproject.toml in the repo root is standard. Defining dependencies in there is standard too. These dependencies are used by different tools (uv, poetry,...) so having ayon-dependencies-tool use them is IMO also standard. If we expect dependencies in some way that is not standard, I would argue that we should change what we expect :)

Having AYON related stuff defined in pyproject.toml has even more advantages - what we do with create_package can be considered as a build backend (PEP5-518) and we could, in time, switch to something like hatch or even create our own and standardize it across the addons.
We can use it for exposing various entry points and many other things. I am side tracking here but my goal is to make an argument for one standard, developer friendly and tool supported, schema validated format that is pyproject.toml

[dee-sometech]

Love your take on this @antirotor <3 ;)

IMHO create_package is a poor man build backend and should totally be replaced by an actual build backend (hatch based plugins are not that complicated to implement...)

[iLLiCiTiT]

Having AYON related stuff defined in pyproject.toml has even more advantages - what we do with create_package can be considered as a build backend (PEP5-518) and we could, in time, switch to something like hatch or even create our own and standardize it across the addons.
We can use it for exposing various entry points and many other things. I am side tracking here but my goal is to make an argument for one standard, developer friendly and tool supported, schema validated format that is pyproject.toml

That is different conversation, but at the end we don't need the pyproject toml in the final zip (on server), we just need what server actually needs in the zip, which is why this PR was created, to store what we need in client.json, with that we can move the dependencies to the root pyproject toml and modify create package script to create the client.json instead.

[martastain]

WDYM by "root pyproject.toml"? Keep in mind that we already use pyproject.toml in the root in multiple repos for server related stuff.

[iLLiCiTiT]

WDYM by "root pyproject.toml"? Keep in mind that we already use pyproject.toml in the root in multiple repos for server related stuff.

You can add something like this to the pyproject toml, does not affect the "standard" pyproject toml (at least that's how I understood it). The point is that you end up with client.json for metadata about addon client instead of pyproject.toml which is used now. How the values would get there is secondary conversation, at least for me.

Example how we could define it in the root pyproject toml:

[ayon.dependencies]
requests="*"

[ayon.runtimeDependencies]
PIL="*"

[antirotor]

Maybe let's talk about how this is currently used - client.json is where you define dependencies for the addon. This gets uploaded to the server so when you want to create dependency package for the given addon bundle, you'll get them. ayon-dependencies-tool will take this, compile pyproject.toml from it and run uv to resolve it.

On the client side, you often need to define the same dependencies twice now (in <addon repo root>/pyproject.toml AND in <addon repo root>/client/pyproject.toml) just to be able to run unit test and other things.

My point is to use just <addon repo root>/pyproject.toml

BTW uv now supports workspaces and they smell like dependency packages....

Note

If there is a need to separate server and client dependencies, we can still use dependency groups - they are there exactly for cases like that.

[iLLiCiTiT]

AND in /client/pyproject.toml) just to be able to run unit test and other things.

This pyproject toml would be removed -> that is kinda the point of this PR to remove that.

you often need to define the same dependencies twice now

Yes, because you want to define more dependencies for running from code which should not be defined in the dependencies for dependencies tool.

My point is to use just /pyproject.toml

I'm not against, but we should not pass the full pyproject to server as server has NO IDEA what to do with it, and dependencies tool either. Also you might want to store more client related metadata in future, which might not be possible with toml.

NOTE: We HAVE TO define the dependencies for dependencies tool in a different way I don't see how it would work if we would not do it in a different way. To change that we'd have to completelly rewrite how we handle dependencies -> TOTALY DIFFERENT CONVERSATION, please do not enter that here. I'm trying to improve what we have now, not what we want in 5 years, if we actually have some time to spend on it, I don't care about 5 years future that might not happen, I want to avoid current confusion.

Remove client/pyproject.toml, instead add client.json to the zip package which contains metadata for client -> allows to add more metadata, reduce the issue with misunderstanding how the pyproject toml should look like, e.g. do not define python version or not define project, or do not use "standard" dependencies (because we can't handle that). How the client.json is defined can wary based on use-case, by default it can look into pre-defined groups in root's pyproject toml. For some time we'll have to create both anyways because of server version compatibility.

Most of addons (90% of them) don't have pyproject.toml in root, that's why I don't think it has to be strictly in the root pyproject toml, because it just does not have to be there most of the time. Again this is current state, I'm not looking for "final solution" I'm looking for something that will not cause the confusion current pyproject toml is causing. The suggested approach allows any approach we'll decide to go with by using a middle man -> client.json, how the data of the file will be filled is up to create package logic, so any addon can customize it up to it's logic. If we decide to go fully with uv and pyproject toml, then we won't need the metadata on server at all and this whole thing will be obsolete.

[antirotor]

I don't want to argue more about this. Just few point and then I am out:

but we should not pass the full pyproject to server as server has NO IDEA what to do with it, and dependencies tool either.

I would argue that you already have that information in pyproject.toml (or should anyway). As you wrote, server has NO IDEA what to do with it - and that's completely fine - it can keep it as it is. Dependencies tool is now parsing the data out and it can do so even with full pyproject.toml with almost no changes in code.

We HAVE TO define the dependencies for dependencies tool in a different.

What different way? We are using uv at the end to get dependencies and it would work just the same, the only difference would be what groups you would use.

Remove client/pyproject.toml, instead add client.json to the zip package which contains metadata for client -> allows to add more metadata, reduce the issue with misunderstanding how the pyproject toml should look like.

I won't reduce confusion that much, you are just replacing one file with another (pyproject with client which is IMO also confusing because it doesn't state what it is, also json doesn't support comments so using it for anything human editable isn't really good option), but the biggest confusion is that everyone expects to define dependencies in the main pyproject.toml

Most of addons (90% of them) don't have pyproject.toml in root, that's why I don't think it has to be strictly in the root pyproject toml, because it just does not have to be there most of the time.

I would argue that not having pyproject.toml is actually antipattern on our side because even making small changes without properly configured linter is just wrong. Correct version of linter is dependency and as such it should be defined in pyproject.toml. Even with addon that are not python based (I am not aware of any) there must be at least a little bit of python to make addon from it and there are tons of successful hybrid projects that are cpp based and yet they still define pyproject.toml.

I know you are not looking for "final solution" and I get why the client/pyproject.toml with it's poetry based nonsense is confusing, but if we are changing how we define addon dependencies anyway, shouldn't we do it more conveniently? I can't really see any way in the future that would allow us to ditch dependency packages that wouldn't use pyproject.toml at the same time.

🫳🏻🎤

[iLLiCiTiT]

I can't really see any way in the future that would allow us to ditch dependency packages that wouldn't use pyproject.toml at the same time.

That's the point. When we ditch dependencies tool we can use pyproject toml. But NOW it is confusing.

What different way? We are using uv at the end to get dependencies and it would work just the same, the only difference would be what groups you would use.

Usually you want to be more strict about versions of dependencies for tests and examples, but for addon compatibility you usually have to be more variable, also you should not re-define dependencies that are already in launcher (like PySide6) or ayon-core (like Pillow). That's what is different. You also should not define python version, but pyproject toml does require it, we need dependencies in key: value, but standard pyproject toml does define it is list of strings. The whole project part in standard pyproject is not used. Server currently does parse the data that dependencies tool uses, it does not return full pyproject toml. And like I've said, the root pyproject toml does not have to be the way how it is defined, that's what you proposed, it is up to the addon's create package how it is passed to the zip package (server).

The point is: It is different, it is not standard, I don't want to change anything except way how it is defined and how it is read by server. It will be always confusing, but having pyproject.toml in client dir causes a tons of confusion. Right now we can't get rid of poetry from the tomls and we can't use the standard dependencies definitions. So we're stuck in the middle of non-standard sh*t that does not represent anything from how it is in fact used.