runme web is to hard to use today. A big problem is there is a lot of functionality much of which is exposed to users by writing JS code which runs in a kernel e.g.
- Configuring using codex
- Adding/manipulating Google Drive Files
- Starting/managing runners
- Connecting using Jupyter
One of the core ways we'd like to make runme easy to use is with AI. Users should just interact with the AI to understand what runme can do and accomplish different tasks.
A core critical user journey(CUJ) is the following
- An existing runme user shares a link to a runme doc
- User opens the link and gets introduced to runme
- User uses the AI to ask questions about the doc
- User sets up a runner in order to be able to execute code
- User uses the AI and the runner to execute some modified version of the doc
- User sets up Jupyter in order to get a REPL
- User uses the AI to generate code that runs inside Jupyter
Since the AI is critical to user's success we'd like to iterate on the prompts and tools to improve the AI's ability to help users use the application.
There are also a number of bugs/features that need to be fixed in order to help make users more successful. For example, there's lot of places where we could do a better job surfacing errors.
I think we can frame this as an RL problem. The CUJ above gives a very small set of tasks that can easily be judged for success
- Open a doc and ask a question
- Create a doc and execute "echo 'Hello World'"
- Use jupyter to plot a sine wave
So I think we could just have an AI try to use runme to accomplish these tasks and then make changes to improve its ability to accomplish these tasks; e.g. by
- Improving prompts
- Improving error reporting and logging
- Adding better tools for the AI to use
To do this we need to setup an environment in which the AI can operate runme and iterate on its code. We are pretty close; we already use CDP to let codex operate runme as part of interactive development (https://github.com/runmedev/web/blob/f108d70e92bc67cd22d991191fecd2bb6d64ec94/docs-dev/chatkit-and-cdp.md).
To make this safe for autonomous we need to setup accounts that the AI can operate with minimal risk
- An isolated gdrive account so the AI can't exfiltrate documents
- An OpenAI API key to use to setup runme
We also need to setup a remote, codex environment which can run the agent. Maybe just an ubuntu VM?
cc @hamelsmu
runme web is to hard to use today. A big problem is there is a lot of functionality much of which is exposed to users by writing JS code which runs in a kernel e.g.
One of the core ways we'd like to make runme easy to use is with AI. Users should just interact with the AI to understand what runme can do and accomplish different tasks.
A core critical user journey(CUJ) is the following
Since the AI is critical to user's success we'd like to iterate on the prompts and tools to improve the AI's ability to help users use the application.
There are also a number of bugs/features that need to be fixed in order to help make users more successful. For example, there's lot of places where we could do a better job surfacing errors.
I think we can frame this as an RL problem. The CUJ above gives a very small set of tasks that can easily be judged for success
So I think we could just have an AI try to use runme to accomplish these tasks and then make changes to improve its ability to accomplish these tasks; e.g. by
To do this we need to setup an environment in which the AI can operate runme and iterate on its code. We are pretty close; we already use CDP to let codex operate runme as part of interactive development (https://github.com/runmedev/web/blob/f108d70e92bc67cd22d991191fecd2bb6d64ec94/docs-dev/chatkit-and-cdp.md).
To make this safe for autonomous we need to setup accounts that the AI can operate with minimal risk
We also need to setup a remote, codex environment which can run the agent. Maybe just an ubuntu VM?
cc @hamelsmu