remake docs by lythx · Pull Request #40 · yaptide/for_developers

lythx · 2026-03-10T12:35:34Z

I created new docs, some pages are rewrites of the old docs, some are new. The base was generated by AI, I reviewed and fixed all the generated pages. There's a few TODOs which I'll resolve before merging.
I used Starlight instead of mkdocs, so the commands are different now:

npm run start - run and test locally
npm run build - build the production website
npm run preview - run the build

Github Pages workflow needs to be tested before merging.

grzanka · 2026-03-10T15:55:14Z

Can you update main README with instruction how to deploy locally ?

Copilot

Pull request overview

This PR migrates the YAPTIDE developer documentation site from a MkDocs/Poetry setup to an Astro + Starlight site, adds a new documentation content structure (architecture/backend/frontend/converter/API reference), and updates CI/dependabot to match the new Node-based build.

Changes:

Replace MkDocs configuration/content with an Astro + Starlight documentation site (new astro.config.mjs, package.json, content under src/content/docs/).
Add extensive new/updated developer documentation pages across setup, architecture, backend, frontend, converter, and API reference.
Update GitHub Pages workflow and Dependabot to use npm + Astro build output (dist/), and adjust repo-level config files (.gitignore, VSCode configs).

Reviewed changes

Copilot reviewed 73 out of 157 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`astro.config.mjs`	Starlight site configuration (site/base/sidebar/logo/edit link).
`package.json`	Introduces Astro/Starlight dependencies and build scripts.
`tsconfig.json`	Adds Astro strict TS config for the docs project.
`src/content.config.ts`	Configures Starlight docs collection loader/schema.
`src/styles/custom.css`	Custom theme tokens + landing-page card/badge styling.
`public/favicon.svg`	Adds SVG favicon with light/dark fill via media query.
`public/assets/logo.svg`	Adds static logo asset for the site.
`src/assets/yaptide-logo.svg`	Adds logo used by Starlight config.
`.gitignore`	Updates ignores for Astro (`dist/`, `.astro/`, `node_modules/`, env files).
`.vscode/launch.json`	Adds VSCode launch config for `astro dev`.
`.vscode/extensions.json`	Recommends Astro VSCode extension.
`.github/workflows/docs.yml`	Switches GitHub Pages build/deploy from MkDocs/Python to Astro/Node (`dist/`).
`.github/dependabot.yml`	Switches dependency updates from pip to npm (+ GitHub Actions).
`src/content/docs/index.mdx`	New developer docs landing page with mode matrix and navigation cards.
`src/content/docs/local-setup/local-frontend-demo.mdx`	Local “demo mode” (frontend-only) setup guide.
`src/content/docs/docker-setup/docker-frontend-demo.mdx`	Docker guide for serving demo build (currently includes TODO placeholders).
`src/content/docs/docker-setup/docker-celery.mdx`	Docker setup guide for backend + Celery workers (currently includes TODO placeholders).
`src/content/docs/docker-setup/docker-slurm.mdx`	Docker SLURM setup stub (currently TODO + tip block).
`src/content/docs/architecture/overview.md`	System-level architecture overview + diagram.
`src/content/docs/architecture/data-flow.md`	End-to-end data flow across direct/batch/local execution paths.
`src/content/docs/architecture/auth-model.md`	Auth model for native vs Keycloak + demo mode behavior.
`src/content/docs/backend/overview.md`	Backend architecture overview (Flask/Celery/Postgres).
`src/content/docs/backend/api-endpoints.md`	Narrative guide to backend endpoints and usage.
`src/content/docs/backend/database.md`	Database schema/model documentation and migration workflow.
`src/content/docs/backend/simulation-lifecycle.md`	Job/task state machine and direct vs batch execution details.
`src/content/docs/backend/simulator-management.md`	Simulator binary management via S3 + encryption.
`src/content/docs/backend/docker-deployment.md`	Docker Compose deployment guide for backend stack.
`src/content/docs/backend/testing.md`	Backend testing approach and fixtures.
`src/content/docs/frontend/overview.md`	Frontend architecture overview and directory structure.
`src/content/docs/frontend/3d-editor.md`	3D editor (Three.js) architecture and core patterns.
`src/content/docs/frontend/simulation-services.md`	Frontend simulation service abstraction and flows.
`src/content/docs/frontend/pyodide-converter.md`	In-browser converter via Pyodide + build pipeline.
`src/content/docs/frontend/geant4-wasm.md`	Geant4 WebAssembly runtime architecture and limitations.
`src/content/docs/frontend/auth-flows.md`	Standard vs Keycloak auth flows + demo mode behavior.
`src/content/docs/frontend/adding-commands.md`	Command-pattern guide for undo/redo and testing.
`src/content/docs/frontend/testing.md`	Frontend test strategy (Jest/RTL) + mocking patterns.
`src/content/docs/converter/overview.md`	Converter purpose, supported engines, and structure.
`src/content/docs/converter/conversion-flow.md`	End-to-end conversion pipeline explanation.
`src/content/docs/converter/adding-a-simulator.md`	How-to guide for adding a new converter engine.
`src/content/docs/converter/shieldhit.md`	SHIELD-HIT12A converter internals and formats.
`src/content/docs/converter/fluka.md`	FLUKA converter internals and card-based format.
`src/content/docs/converter/geant4.md`	Geant4 converter internals (GDML + macro).
`src/content/docs/converter/testing.md`	Converter testing strategy (golden files + unit tests).
`src/content/docs/contributing/guide.md`	Contribution workflow across repos.
`src/content/docs/contributing/code-style.md`	Style/lint conventions across Python/TS repos.
`src/content/docs/contributing/glossary.md`	Project glossary for domain and architecture terms.
`src/content/docs/api-reference/overview.md`	API reference overview + endpoint grouping.
`src/content/docs/api-reference/auth.md`	Auth endpoint reference.
`src/content/docs/api-reference/jobs.md`	Jobs endpoint reference.
`src/content/docs/api-reference/results.md`	Results/inputs/logs endpoint reference.
`src/content/docs/api-reference/user.md`	User/clusters endpoint reference.
`mkdocs.yml`	Removes MkDocs site config.
`pyproject.toml`	Removes Poetry/MkDocs dependency definition.
`docs/index.md`	Removes old MkDocs landing stub.
`docs/gen_ref_pages.py`	Removes MkDocs auto-reference generator script.
`docs/backend/*`	Removes old MkDocs backend docs.
`docs/frontend/*`	Removes old MkDocs frontend docs.
`docs/converter/*`	Removes old MkDocs converter docs.
`docs/documentation/index.md`	Removes old MkDocs “how docs work” page.

Comments suppressed due to low confidence (1)

.github/workflows/docs.yml:56

actions/configure-pages runs for pull_request events as well, but PRs (especially from forks) typically don't have GitHub Pages write permissions. Consider gating Pages-specific steps (configure-pages, upload-pages-artifact, deploy-pages) behind the same push && refs/heads/main condition, and keep PR runs to just npm ci/npm run build to avoid CI failures.

      - name: Setup Pages
        uses: actions/configure-pages@v5
      - name: Upload artifact if on main
        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
        uses: actions/upload-pages-artifact@v4
        with:
          path: 'dist'
      - name: Deploy to GitHub Pages if on main
        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
        id: deployment
        uses: actions/deploy-pages@v4

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-10T16:03:52Z

src/content/docs/docker-setup/docker-slurm.mdx

+:::tip
+  Prefer a local setup? See [Local — SLURM](/docs/local-setup/local-slurm/) for the non-containerized version.
+:::                                                          


The admonition block appears to be closed with ::: followed by trailing whitespace. Starlight/markdown-it container syntax typically expects the closing fence to be exactly :::; the extra spaces can prevent the tip block from closing/rendering correctly. Remove the trailing whitespace so the closing line is just :::.

src/content/docs/index.mdx

Copilot · 2026-03-10T16:03:53Z

astro.config.mjs

+            }
+          ],
+        },
+          {


The sidebar config for the "Docker Setup" section has inconsistent indentation (the opening { is indented differently than the rest of the array items). While it still parses, it reads like an accidental nesting/copy-paste error and makes future edits risky. Reformat this block (e.g., run the repo formatter) so the sidebar array items are consistently aligned.

Suggested change

{

{

Copilot · 2026-03-10T16:03:53Z

src/content/docs/docker-setup/docker-frontend-demo.mdx

+## Prerequisites
+
+TODO Docker setup should not require Node.js or Python - change the dockerfile and update this guide
+
+- **Node.js 20+** with npm
+- **Python 3.9+** with pip and venv — needed to build the converter into a Pyodide/WebAssembly module (only if building locally)
+- **Docker**


There is a TODO placeholder in the published docs. If this page is meant to be user-facing, replace the TODO with actual prerequisites (or move it into an explicit "Known limitations" callout) so the guide is actionable.

src/content/docs/docker-setup/docker-celery.mdx

Copilot · 2026-03-10T16:03:54Z

src/content/docs/docker-setup/docker-celery.mdx

+## Step 3: Start the frontend
+
+TODO change this to use Docker instructions — the frontend currently requires a local setup
+
+The backend API is available at **http://localhost:5000** (via NGINX). Start the frontend in a separate terminal:
+```bash


This TODO indicates the core instructions are not yet Docker-based ("change this to use Docker instructions"). If the page is going into the sidebar/navigation, consider either completing this section or adding an explicit note that the Docker setup is currently backend-only and the frontend must be run locally, to avoid misleading readers.

grzanka

See comments inline

grzanka · 2026-03-11T10:07:17Z

README.md

+```
+
+2. **Install dependencies**
+```bash


its a very bad practice to put two commands in one copy-pasteable block

README.md

grzanka · 2026-03-11T10:11:59Z

src/content/docs/architecture/overview.md

+
+## Architecture Diagram
+
+```


maybe this could be converted to mermaid or something better looking ?

src/content/docs/architecture/overview.md

grzanka · 2026-03-11T10:16:04Z

src/content/docs/local-setup/local-frontend-demo.mdx

+## Setup
+
+```bash
+git clone https://github.com/yaptide/ui.git


Avoid everywhere putting multiline multiple commands into single copy-pasteable block. If I copy-paste I would like to do it command-by-command. everywhere.

src/content/docs/local-setup/local-celery.mdx

grzanka · 2026-03-11T10:26:40Z

src/content/docs/architecture/auth-model.md

+│    UI    │                           │ Backend │
+└────┬─────┘                           └────┬────┘
+     │                                      │
+     │  PUT /auth/register                  │


Do we have such endpoint ? I always believed that user registration was disabled by purpose as architecture desgin decision (to be document here!).
We only allow registration by command line script.

Local users (where we store username and password) are only required for the development mode.

On production we don't store user passwords, we only use keycloak.

Maybe we should have somewhere a documentation on decisions made ?

grzanka · 2026-03-11T10:27:08Z

src/content/docs/architecture/auth-model.md

+
+| Token | Lifetime | Storage |
+|---|---|---|
+| Access token | 10 minutes | httpOnly cookie |


Have you confirmed these numbers ?
Are they specific to the cyfronet keycloak instance ?

grzanka · 2026-03-11T10:29:53Z

src/content/docs/architecture/auth-model.md

+When a Keycloak token arrives, the backend checks the `PLG_YAPTIDE_ACCESS` claim in the token. This ensures the user has been granted access to the YAPTIDE service in the PLGrid infrastructure.
+
+The backend also:
+1. Fetches **SSH certificates** from a dedicated cert-auth service (`CERT_AUTH_URL`)


we shouldn't be confiused with SSH public-private key pair here. Those are short lived certificatates. Maybe we should point the developer to some documentation of that ?

Maybe you find something better than this:
https://goteleport.com/blog/how-to-configure-ssh-certificate-based-authentication/

src/content/docs/backend/api-endpoints.md

… instructions

grzanka · 2026-03-12T18:55:23Z

.github/workflows/docs.yml

-        run: |
-          poetry lock
-          poetry install
+          node-version: '20'


why not v24 ?

see https://nodejs.org/en/about/previous-releases

Copilot

Pull request overview

Copilot reviewed 73 out of 80 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-12T19:08:14Z

README.md

 # Yaptide developer documentation
-https://yaptide.github.io/for_developers/
+
+[](https://yaptide.github.io/for_developers/)


README contains an empty Markdown link ([](https://yaptide.github.io/for_developers/)), which renders as a blank clickable area. Replace it with a proper link label (or plain URL) so the landing link is visible/readable.

Suggested change

[](https://yaptide.github.io/for_developers/)

[Yaptide developer site](https://yaptide.github.io/for_developers/)

Copilot · 2026-03-12T19:08:14Z

src/content/docs/local-setup/local-slurm.mdx

+description: Run the YAPTIDE backend locally with Keycloak authentication for SLURM integration development.
+---
+
+import { Tabs, TabItem } from "@astrojs/starlight/components";


Tabs/TabItem are imported but never used in this MDX file. With strict TS settings, unused imports can cause type-check failures and should be removed (or add the corresponding <Tabs> usage if intended).

Copilot · 2026-03-12T19:08:14Z

src/content/docs/docker-setup/docker-celery.mdx

+description: Run the full YAPTIDE backend with Celery workers in Docker containers for integration testing and development.
+---
+
+import { Tabs, TabItem, Aside } from "@astrojs/starlight/components";


Aside is imported but not used anywhere in this page. Please remove it from the import list to avoid unused-import/type-check issues.

Copilot · 2026-03-12T19:08:15Z

src/content/docs/backend/api-reference.mdx

+import SwaggerUI from 'swagger-ui-react'
+import 'swagger-ui-react/swagger-ui.css'
+
+<SwaggerUI client:only="react" url="/for_developers/openapi.yaml" />


The Swagger spec URL is hard-coded to /for_developers/openapi.yaml, which couples this page to a specific base path. Use the runtime base (e.g. import.meta.env.BASE_URL) to build the URL so it keeps working if the site base changes or if this page is reused under a different deployment path.

Copilot · 2026-03-12T19:08:15Z

src/content/docs/contributing/guide.md

+git push origin feature/my-change
+```
+
+Open a pull request on GitHub against `master`. Include:


This guide says to open PRs against master, but this repository/workflow is clearly using main as the default branch (e.g. Pages workflow triggers on main). Please update the text to avoid sending contributors to the wrong target branch.

remake docs

dfd526d

lythx had a problem deploying to github-pages March 10, 2026 12:35 — with GitHub Actions Failure

grzanka assigned lythx Mar 10, 2026

grzanka requested a review from Copilot March 10, 2026 15:53

Copilot started reviewing on behalf of grzanka March 10, 2026 15:54 View session

Copilot AI reviewed Mar 10, 2026

View reviewed changes

lythx added 4 commits March 10, 2026 19:04

move contributing up in the sidebar order and make it open by default

e355381

fix problems in setup docs

219f53c

remove unneeded sections and improve contributing docs

e252f13

remove json schema todo

a70d9ff

lythx had a problem deploying to github-pages March 10, 2026 18:05 — with GitHub Actions Failure

lythx added 7 commits March 10, 2026 19:12

fix astro config indent

40d7fdc

fix docker slurm guide whitespaces after :::

01fb65f

change base url from /docs to /for_developers

5a2cadb

fix base path and add favicon

9391216

add setup instructions in readme

cbfe2ba

add a 404 page to fix the gh actions fail

b9be01c

remove unneeded public folder

33a97b5

lythx had a problem deploying to github-pages March 10, 2026 18:34 — with GitHub Actions Failure

lythx added 2 commits March 10, 2026 19:45

change remaining TODOs to informational Work In Progress components

51a5335

fix GHA build error by disabling 404 page check

6afb6b9

lythx had a problem deploying to github-pages March 10, 2026 18:46 — with GitHub Actions Failure

fix GH Actions error by overriding zod version

5b0f0a8

lythx had a problem deploying to github-pages March 10, 2026 18:53 — with GitHub Actions Failure

fix GH Actions build fail by downgrading astrojs

8f81f71

lythx temporarily deployed to github-pages March 10, 2026 18:57 — with GitHub Actions Inactive

fix name and version in package.json

b94d41e

lythx temporarily deployed to github-pages March 10, 2026 18:59 — with GitHub Actions Inactive

change front docker instructions to use docker compose

385e060

move favicon to public/ dir

7b4ed6d

lythx temporarily deployed to github-pages March 11, 2026 09:21 — with GitHub Actions Inactive

add submodule update command to docker front setups

e0ba5cd

lythx temporarily deployed to github-pages March 11, 2026 09:29 — with GitHub Actions Inactive

fix localhost port in docker celery tutorial

91db2d3

lythx temporarily deployed to github-pages March 11, 2026 09:39 — with GitHub Actions Inactive

grzanka requested changes Mar 11, 2026

View reviewed changes

disable astro telemetry in GHA

50b1b36

lythx temporarily deployed to github-pages March 12, 2026 08:06 — with GitHub Actions Inactive

lythx added 4 commits March 12, 2026 09:54

remove multi-command copypaste blocks and improve comments in startup…

d71c473

… instructions

change multiline commands to singleline in local celery setup

4158498

add information about tradeoffs of docker vs local setup

338d605

remove dependency versions from architecture overview

a011c13

lythx temporarily deployed to github-pages March 12, 2026 10:10 — with GitHub Actions Inactive

lythx added 3 commits March 12, 2026 18:10

remove block with multiple commands from readme

51edb23

add detailed explanation for the download-shieldhit command

c98b8fa

use swagger in api reference page

b969ab5

lythx temporarily deployed to github-pages March 12, 2026 18:50 — with GitHub Actions Inactive

grzanka reviewed Mar 12, 2026

View reviewed changes

grzanka requested a review from Copilot March 12, 2026 19:03

Copilot started reviewing on behalf of grzanka March 12, 2026 19:03 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

lythx added 4 commits March 12, 2026 21:30

change node version in github workflow to node 24

6f9254b

fix readme yaptide docs link

ec1939b

remove unused imports

aeed455

clarify branch names for different repos in contributing guide

d278d06

lythx temporarily deployed to github-pages March 12, 2026 20:40 — with GitHub Actions Inactive

grzanka approved these changes Mar 12, 2026

View reviewed changes

grzanka merged commit ad157fb into main Mar 12, 2026
3 checks passed

grzanka deleted the remake-docs branch March 12, 2026 20:45

	[](https://yaptide.github.io/for_developers/)
	[Yaptide developer site](https://yaptide.github.io/for_developers/)

Conversation

lythx commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grzanka commented Mar 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

grzanka left a comment

Choose a reason for hiding this comment

Uh oh!

grzanka Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

grzanka Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

grzanka Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

grzanka Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

grzanka Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

grzanka Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

grzanka Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

lythx commented Mar 10, 2026 •

edited

Loading