Analysis: cue cmd v2: cue cmd analysis

[myitcv]

For many users, cue cmd is where CUE really "clicks": declarative workflows that act as the perfect glue between data, logic, control flow, and code.

We’ve been gathering feedback, bugs, and feature requests for a potential v2 of cue cmd in issue #1325 and linked issues/discussions.

We have now prepared an analysis to move forward the discussion:

https://github.com/cue-lang/proposal/tree/main/analyses/4081-cue-cmd.md

Please use this discussion to provide feedback on this analysis.

[tvandinther]

I read the point about using lists as implicit ordering but didn't spot a point about using lists as unnamed tasks (with the same semantics as now with parallelism unless dependent).

Given these 2 options are mutually exclusive and my preference for the latter, I want to put it on record.

[the-nurk]

would associative lists not allow fixes for this semantic separation?

[infogulch]

I don't see why naming tasks is a big deal. Using the index as the key in a struct comprehension is hardly an inconvenience compared to a hypothetical list comprehension for the same thing: (play)

{ for i, dir in dirs { ("\(i)"): exec.Run & { cmd: ["ls", dir] } } }

[ for dir in dirs { exec.Run & { cmd: ["ls", dir] } } ]

By comparison, manually threading $after and $done fields for every task in a sequence is quite cumbersome, especially if tasks are nested.

Here xtemplate/make_tool.cue:task.dist I create a comprehension that declares a six supertasks that can run in parallel and don't need specific names, and within each supertask there are a bunch of subtasks that need to execute in a linear sequence. This is an interesting case because both mutually exclusive meanings for lists of tasks could help here, but personally I would much prefer the one where I wouldn't have to explicitly sequence a long chain of dependent tasks. I confess this last statement is a fib, I actually do need them to have names because I require them all to be sequenced after the testing stage of the ci pipeline when I reuse this later in the file. (Only a fib because previous version of this command didn't use their names and it was still nbd.)

I'm coming around to Paul's point that maybe lists are not worth it for either case because referencing tasks is very common (and becomes more likely as command complexity grows), the cost of adding an unneeded (for now) dummy name is low, and if task dependencies are easier to define then maybe that use for lists evaporates too.

[tvandinther]

I agree that the workaround is easy, but for me it is about consistency. I think it comes down to defining a concrete structure on how workflows are defined. My intuition when writing a _tool.cue file is that whenever I create a struct using a tool call (e.g. exec.Run), that task will be found and put into an execution DAG (see Roger's example here). With this assumption, defining a list of these structs should mean they are pieced into the DAG through this same discovery mechanism. Dependencies are the explicit aspect here which based on this mental model, would need to be defined.

For me its about defining a model that everyone can intuitively adopt which stays consistent when writing these workflows.

[infogulch]

I understand your point that there are two kinds of datastructures in CUE, structs and lists, and it would make sense for CUE cmd to treat them similarly.

I'm just not sure if it's worth it to lose references.

[infogulch]

The analysis could list projects that use CUE for build system related tooling. Some of these are new to me:

• infogulch/xtemplate:make_tool.cue a project CI built directly with cue cmd
• marxarelli/masse "A CUE based BuildKit frontend"
• hof (does this count?)
• CUE was supported in the early days of the Dagger CI system. Ending Support for the Dagger CUE SDK (2023) has a lot of links.

[infogulch]

It would be helpful if the analysis went into some detail regarding how cue cmd currently executes.

• How cue finds and queues up new tasks to execute
• How task completion state is mixed back into the cue cmd state
• Terminology for the above ideas
• A worked example explaining what happens during execution of a simple cue cmd run. Taken down to the level of what Go code runs in what order. The hello world example from the office hours slides would be a good candidate.
• Where the $O(n^2)$ or $O(n^3)$ performance cliffs are and why
  • A minimal example of the performance issue. (It should not be hard to find a $O(n^3)$ cliff!)
  • Plans to address these, even vague

[infogulch]

As mentioned in the first cue cmd v2: office hours #4080, the task/command terminology has multiple overlapping definitions which makes communication about concepts in cue cmd difficult.

Here are some of the multiple meanings of these terms that I could come up with:

• "task"
  • A cue cmd tool call definition, e.g. cli.Print & { text: "abc" }
  • A group of tool calls
  • The broad idea of a thing the user wants to do
• "command"
  • The cue cmd system, spoken like "cue command"
  • Any executable
  • The exec.Run tool
  • A group of tool calls
  • A group of tool calls named in a *_tool.cue file under command: xyz: ...
  • Invoking a named group of tool calls with the cli like cue cmd xyz
  • the broad idea of a thing the user wants the computer to do

We should discuss various terminology options and decide on some precise definitions. Even if they're flawed and they need to change in the future.

[infogulch]

Here's an idea:

• tool call is pretty clear already, so keep it
• task: one or more tool calls and tasks
• leaf task: an alias for a single tool call
• subtask: one or more tool calls, with the implication that it is part of a larger task with other subtasks

Command is a bit trickier, maybe we need some synonyms.

[tvandinther]

I see the command as a binding between a task and the command line. The task is the real entrypoint and can be a single task or multiple tasks. Ideally tasks could be recursively defined a little bit like they are now but outside of the scope of a command.

Then the command is like I mentioned, just a binding of a task to cue cmd.

[infogulch]

I like that phrase, "binding the cue cli invocation to a task".

Then the issue is how to disambiguate other incidental uses of the term "command".

Maybe the system should be called cue run. I think this was mentioned before somewhere. So you would invoke it in the cli with cue run mycommand in the cli that corresponds to the cue definition command: mycommand: { ... }. So the system is a "command runner", and it aligns with other cli runner terminology like npm run ....

With this clear correspondence between the cli invocation and the command definition, renaming the system to "cue run", and being a bit careful about using "exec"/"execute" terminology for exec.Run instead of "command", I think the ambiguity problem would be mostly resolved.

[infogulch]

It would be nice if the analysis expanded on the idea of carving out a smaller more capable core of cue cmd and reimplementing other cue functionality (cue export, cue vet, etc) on top. This was mentioned briefly in this first version (search for "A bullet point"), and was also discussed in the 4080 office hours.