Adding Control Flow Functionality to Workflows

[jackpeplinski]

Current Limitations

Currently, workflow steps are recorded in JSON. While JSON is a convenient data exchange format, it lacks native control flow functionality, making it difficult to express common patterns such as:

• Iterating over multiple elements (e.g., clicking a download button for every file on a page)
• Conditional execution (e.g., only click if an element exists)

Options

Option: AI-Generated Code
While generating code and just limiting the functions accessible would be optimal in my view, this quickly becomes complex since no current LLMs verify the generated code, AFAIK. e.g., If I prompt "Only use functions A, B, C... while writing your code" there's no guarantee these functions 1) aren't in the code and 2) that the code is structured as I intend.

Option: JSON-Based
An alternative is to add a few common control flow steps to the JSON schema since many LLMs do provide structured outputs, which can then be transformed into Python code. Users could modify either the JSON or the generated Python, depending on their preferences. This also allows for more control when a workflow pattern cannot currently be expressed in the JSON schema.

Implementation

Proposed Initial JSON Extensions

• ForLoopStep / WhileLoopStep for iteration
• ConditionalStep for if_exists and other conditional logic
• Loop variables ({index}, {item}) and scoping for dynamic execution

Implementation Outline

1. Extended JSON Schema – Add new step types (ForLoopStep, WhileLoopStep, ConditionalStep) in views.py
2. JSON-to-Python Code Generator – Convert control flow JSON to executable Python with proper indentation in workflows/workflow_use/codegen/generator.py
3. Workflow Execution Engine – Extend Workflow class in service.py to handle loops and context
4. Builder Service Updates – Update LLM prompts to generate control flow structures in prompts.py
5. Context Management – Support loop variables and scoping in service.py

Conclusion

This approach keeps the structured JSON format while enabling richer workflow logic. Let me know your thoughts.

Note
P.S., While exploring this idea, I noticed Skyvern is pursuing a similar idea, though they currently lack a browser extension recording setup and I find their documentation less up-to-date.

[nnWhisperer]

Hi, I found here while searching a deterministic execution of browser-use interactions. Why invent a new language while you could use existing building blocks, like python? What matters is what api will be exposed, becoming the functions that could be called using the python code. Playwright already did this and abstracted browser interaction into an api. Hence, why not arriving at creating playwright scripts from the browser-use runs? One could generate these using browser interaction history, in a consistent manner rather than asking LLM to write them.

[jackpeplinski]

I find your message a bit hard to read. Could you run it through an LLM and rewrite it?

[nnWhisperer]

Why not create playwright scripts from the browser-use runs? One could generate these using browser interaction history, in a consistent manner rather than asking LLM to write them.

[nnWhisperer]

I'm criticizing the use of new language structures proposed here, such as adding a new forloopstep etc. Then I ask, why invent a new language, while you could reuse existing building blocks, like in python? Then, I proceed by further exemplifying Playwright: already the needed concepts about browser interaction are broken down by Microsoft's playwright team. Rather than reinventing the square wheel, I'm suggesting to be stuck using the classics, like Playwright.

[MagMueller]

What do you think of Jupyter notebook like format. So you can run the workflow step by step and edit it? I can launch this next week.

[jackpeplinski]

That would be cool to see!