Jupyter notebooks integration in Testground

[raulk]

Context

The gossipsub security hardening project created a bespoke Jupyter notebooks integration that allowed users to execute test runs on cluster:k8s and analyse the outputs from within the comfort of an interactive Jupypter notebook session. It was hugely useful and successful.

We need the same for Project Oni.

Output

• Generalise the solution, decouple it from the gossipsub project, package it so that it's project-agnostic and reusable.
• Contribute it to testground/testground.
• Based on conversations with Yusef, the main point to figure out is how to automatically translate the test parameters as expressed in the manifest schema into Jupyter widgets.
  • One proposal was to use a JSON schema => Jupyter Widgets library, by previously translating our super-simple manifest definitions to basic JSON schema (we really want to stay away from JSON schema in Testground).
• Allow the user to easily process test outputs from within Jupyter Notebooks.
• Tackle default parameters for composition groups.
• For cluster:k8s, mount NFS into Jupyter Notebooks server so we can skip the collect outputs step.

Future

• pandas workers.

[raulk]

@yusefnapora ^^ scoping by groups.

[yusefnapora]

just dropping a link to Anton's draft infra PR: testground/infra#56

[yusefnapora]

I've been prototyping this using the holoviz Param library, and have some basic UI generation working.

Here's what I think we need to get working to make this functional:

• support loading an existing composition file
• ability to set params for each composition group, based on param definitions in manifest
• distinguish params visually based on scope/applies-to field
• save composition file based on UI
  • must not destructively alter unknown fields from original composition (e.g. if the UI doesn't support a field, it shouldn't remove it from the output when saving)

Things that are nice to have but may get pushed to next week:

• generate new composition from scratch
  • this is what I have so far in the prototype, but is less important than modifying an existing composition and will likely need rewriting
• dynamically show/hide params based on scope

There's also the "other half" of this story, which is collecting and processing test outputs; I'll open a new issue to track that soon.

[yusefnapora]

Also, I've been working with a local "scrach" repo; where should this code actually live? Will it belong in the main testground repo, or should we make it a standalone tool? @raulk ?

[nonsense]

@yusefnapora any chance we can see a demo of this today or on Friday? I think it will help if we start small with the UI, integrate it with the infrastructure, and continue from there to refine and add features.

I can't say where the code should live, because I am not sure how much of it is testplan specific, and how much of it should be deployed alongside the infra stuff (Jupyter Hub, etc.) - I guess we can decide when we see the first iteration in action? WDYT?

[raulk]

@yusefnapora it would be awesome if you could demo a sneak peek of this today (not excluding a demo on Friday!), as well as walking us through the codebase. Then @nonsense and I can decide where these components will be hosted.

[raulk]

@yusefnapora as discussed yesterday, outstanding items for this sprint include:

• Ability to create a new composition from within the tooling end-to-end, without having to put together a strawman composition file manually first.
• Collecting outputs for analysis the naïve way (Jupyter running locally, against a remote TG daemon, downloading the outputs bundle).
• Run Jupyter notebook reliably locally, against a local or remote TG daemon.

Next sprint:

• Figure out Jupyter client/server architecture so we can send computation off to a Jupyter instance running inside the TG k8s cluster so we can mount EFS directly and perform computation colocated with the data.
• Scopes.

Anything that I'm missing?

[yusefnapora]

@raulk I think that basically covers it.

I think that "run Jupyter notebook reliably locally" should include some kind of Docker magic to avoid dep hell.

I spun my wheels a bit on that yesterday - it's more annoying than I thought, because you can't really bind mount $TESTGROUND_HOME into a container and access plan manifests that were symlink'd in with testground plan import, since the symlinks aren't traversable inside the container.

So I may punt on that for right now and require a python virtualenv to run locally, or just force the user to manually provide the manifest file. The latter might be simpler and less "magic" anyhow, so maybe we should just do that.

[nonsense]

I think that "run Jupyter notebook reliably locally" should include some kind of Docker magic to avoid dep hell.

We should definitely do that - Docker is more-or-less a dependency for Testground today, so no reason to not use it and have users install manually Python and related dependencies.