Reorganize the tutorials before v1.0 by eduardojsbarroso · Pull Request #45 · LSSTDESC/CLPipe

eduardojsbarroso · 2026-11-06T11:29:45Z

Last PR before the v1.0.0 release. The tutorials folder now explains what is in CLPipe, how to run an analysis, how each step works, and how to add a stage.

tutorials/README.md is the entry point. It shows the pipeline, which tutorial covers which step, and how to get the Jupyter kernel at CC-IN2P3 and NERSC. All the notebooks run with the shared firecrown_clp kernel, which starts the firecrown_developer_clp environment.

New pages

running_an_analysis.md: the files of a run directory, the job script, the outputs, how to rerun part of a run, and examples/template for new runs.
txpipe.md: what CLPipe uses from TXPipe and where to find it. The stages are documented in TXPipe, so this is only a page, pointing to the clpipe-paper-v1 tag used for the paper.
covariance_tjpcov.ipynb: runs CLPCovariance on the TXPipe SACC file of the redMaPPer baseline analysis and shows the counts covariance. It reproduces the covariance of the analysis.
likelihood_firecrown.ipynb: writes the CLPFirecrown files, shows what they contain, and evaluates the likelihood once with the CosmoSIS test sampler.
theory_prediction.ipynb: the tutorial of Update cluster_theory_pred to match CLPFirecrown and add a tutorial #40, moved.
writing_a_stage.ipynb: a toy stage that removes the lowest richness bins of a SACC file, how to use it in a pipeline, and the steps to add a stage to CLPipe (file, __init__.py, tests, docs).

The notebooks share their inputs in tutorials/data (the redMaPPer baseline SACC files before and after the covariance, and the cosmoDC2 fiducial cosmology) and write to tutorials/outputs, which is ignored by git. They are saved with their outputs. The old firecrown_tutorial (with its committed CosmoSIS outputs) and txpipe_tutorial folders are removed.

Other changes

Version 1.0.0, in pyproject.toml and in the documentation.
The tutorials are part of the documentation, rendered with myst-nb without running them. docs/conf.py copies them at build time, so there is a single version in git.
firecrown_environment.yml now pins Firecrown (commit 7098afc, 1.15.0a1.dev187), CROW (commit 8a79d03) and TJPCov (0.6.0, from PyPI instead of the clustercount_cov_validation branch, it contains TJPCov #113). The versions of the CLPipe paper for the other packages are pinned in Pin the Firecrown environment to the versions of the CLPipe paper #46.

At NERSC, the shared firecrown_clp kernel now starts firecrown_developer_clp, as at CC-IN2P3, and that environment was updated to the same firecrown, crow and TJPCov. The four notebooks run at both sites with the same results.

tutorials/README.md now describes the pipeline, which tutorial covers which step, and the shared environments and Jupyter kernels at CC-IN2P3 and NERSC. TJPCov 0.6.0 includes the cluster count covariance (PR #113), so the Firecrown environment installs it from PyPI instead of the merged branch, and the README no longer lists TJPCov with TXPipe.

… kernel

codecov-commenter · 2026-11-06T11:38:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.12%. Comparing base (6b4cb17) to head (8cdf52c).

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #45   +/-   ##
=======================================
  Coverage   95.12%   95.12%           
=======================================
  Files           9        9           
  Lines        1025     1025           
  Branches       69       69           
=======================================
  Hits          975      975           
  Misses         41       41           
  Partials        9        9

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…CI job

…covariance element The covariance notebook runs the stage without capturing its output, the likelihood notebook calls cosmosis with a shell line, and the stage notebook writes its output under the temporary name of ceci and calls finalize.

…ep CLPCovariance unchanged

… kernel

…ject.toml

…rsions of the shared environments

jeanjacquemier · 2026-11-12T08:54:47Z

All the notebooks have been executed, without error locally (Ububtu), at CCIN2P3 and NERSC with the "shared" jupyter kernels.

jeanjacquemier · 2026-11-10T10:15:31Z

+    launch_job_all_in2p3.sh   job script that runs everything
+```
+
+The pipeline is split in three files because TXPipe and the CLPipe stages run


This text could benefit from more detail or being moved. As it stands, it lacks clarity.

At the point when the user begins the tutorial, the TXPipe and CLPipe steps have not yet been defined. It would be helpful, at the very least, to indicate which steps correspond to which files.

OR

Move the running_an_anlaysis to the end of tutorial tree, as all the missing information are indeed present in the other chapter.s

jeanjacquemier · 2026-11-10T10:51:35Z

+export PYTHONPATH=/path/to/CLPipe:$PYTHONPATH
+ceci TJPCov.yml
+ceci Firecrown.yml
+


Stages defined in TJPCov.yml and Firecrown.yml are executed in the same environment (firecrown_developer_clp) and are implemented in the same module: clpipe.
So why are they defined in two deference ceci pipeline files (*.yml)?

Because at some point, they were not able to run on the same envs. After some changes to Firecrown and TJPCov, yes. But I decided to keep it like this now just in case something breaks in the near future. People still did not decide where to run TJPCov. Like what we just did or inside Firecrown. So let's leave this to a future problem

Ok, it makes sens.

jeanjacquemier · 2026-11-10T11:51:50Z

+```
+
+The `PYTHONPATH` lines point to the CLPipe and TXPipe checkouts used by the
+run. The last line samples the likelihood with the files written by


replace written by generated to emphasize the fact that this stage generates a new python file.

jeanjacquemier · 2026-11-10T11:53:01Z

+mpirun -n ${SLURM_NTASKS} cosmosis --mpi sampler_file.ini
+```
+
+The `PYTHONPATH` lines point to the CLPipe and TXPipe checkouts used by the


The new installation procedure defined in the README.md include a clpipe module installation in the conda env. So there is no more need to add clpipe path to PYTHONPATH

jeanjacquemier · 2026-11-10T11:59:17Z

+the next steps start from the SACC file they write.
+
+The code and the documentation of these stages are in TXPipe
+(https://txpipe.readthedocs.io). This page only says what CLPipe uses and


Change the url to an hyperlink: TXPipe doc

jeanjacquemier · 2026-11-10T13:06:25Z

This notebook can be executed without any error (locally/CCIN2P3 and NERSC).
I did not understand what the figures/plots show, but that is likely due to my lack of knowledge on the subject.

jeanjacquemier · 2026-11-10T13:22:06Z

This notebook can be executed without any error.

jeanjacquemier · 2026-11-10T13:30:03Z

This notebook can be excuted without any error (locally/CCIN2P3 and NERSC)..

jeanjacquemier · 2026-11-10T14:35:18Z

+```bash
+mkdir -p ${HOME}/.local/share/jupyter/kernels
+cp -r /sps/lsst/groups/clusters/cl_pipeline_project/jupyter_kernel/firecrown_clp ${HOME}/.local/share/jupyter/kernels/
 ```


At least inform the user that the kernel.json must be modifying to replace ${HOME} by the user home directory

I will update the file on CCIN2P3 so nothing has to be done here

…omments The tutorials are numbered in reading order, and running an analysis comes after the four step tutorials, which define its stages and files. The writing-a-stage notebook now shows the config.yml file and the data path for running the stage from the command line. CLPFirecrown files are described as generated, the TXPipe documentation is a link, and the run page says when the CLPipe path is needed in PYTHONPATH.

eduardojsbarroso added 6 commits November 6, 2026 11:57

Reorganize the tutorials (work in progress)

5ddac75

Add the tutorials to the documentation, and final text of the notebooks

6c24b85

Covariance tutorial: remove the files of a previous run

5c3ceb7

Stage tutorial: print the names of the removed richness bins

860b606

Tutorial notebooks executed at CC-IN2P3 with the shared firecrown_clp…

69f4894

… kernel

eduardojsbarroso added 5 commits November 6, 2026 13:17

Version 1.0.0, and myst-nb in the Firecrown environment for the docs …

6e58a50

…CI job

Covariance tutorial: hide the output of TJPCov with %%capture, and ke…

46cfa96

…ep CLPCovariance unchanged

Covariance tutorial: fix the sentence about %%capture

0a5d7c8

Tutorial notebooks executed at CC-IN2P3 with the shared firecrown_clp…

e54e695

… kernel

eduardojsbarroso requested a review from jeanjacquemier November 6, 2026 15:16

eduardojsbarroso added 2 commits November 6, 2026 16:20

Link the documentation at lsstdesc.org/CLPipe in the README and pypro…

c33cd28

…ject.toml

Pin firecrown, crow and TJPCov in the Firecrown environment to the ve…

0740366

…rsions of the shared environments

jeanjacquemier reviewed Nov 12, 2026

View reviewed changes

eduardojsbarroso merged commit 07656a8 into main Nov 12, 2026
6 checks passed

eduardojsbarroso deleted the tutorials branch November 12, 2026 10:16

eduardojsbarroso mentioned this pull request Nov 12, 2026

Pin the Firecrown environment to the versions of the CLPipe paper #46

Merged

Conversation

eduardojsbarroso commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jeanjacquemier commented Nov 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

eduardojsbarroso commented Nov 6, 2026 •

edited

Loading

codecov-commenter commented Nov 6, 2026 •

edited

Loading