CLPipe

CLPipe is the current implementation of the LSST DESC Cluster Pipeline, developed by the Cluster Working Group of the LSST Dark Energy Science Collaboration (DESC) to prepare cosmological analyses of galaxy clusters with Rubin LSST data. This repository does not reimplement the analysis tools: it runs the existing DESC tools together, in a single automated and reproducible chain, from galaxy and cluster catalogs to cosmological parameter constraints. TXPipe measures the cluster abundance and the stacked weak lensing profiles, TJPCov computes the theoretical covariance of the cluster counts, and Firecrown builds the likelihood, with theory predictions from CROW (built on CCL and CLMM), which CosmoSIS then samples.

The stages are connected with ceci through their input and output files, so a stage can be replaced or extended without changing the rest of the pipeline. The current version runs a binned analysis of cluster counts and stacked weak lensing profiles, and has been validated on mock data and on the cosmoDC2 redMaPPer cluster catalog.

The documentation is at https://lsstdesc.org/CLPipe/. It includes the tutorials of the tutorials folder, which show how to run an analysis, how each step works, and how to add a stage.

Installation

To run the examples, there is no need to create new Conda environments — shared environments are already available on both IN2P3 and NERSC.

Because some Python packages required for Firecrown are incompatible with those required for TXPipe, two environments are provided:

• Firecrown — for the covariance (TJPCov) and the MCMC inference
• TXPipe — for the measurements

Firecrown

# IN2P3
conda activate /sps/lsst/groups/clusters/cl_pipeline_project/conda_envs/firecrown_developer_clp

# NERSC
conda activate /global/cfs/projectdirs/lsst/groups/CL/cl_pipeline_project/conda_envs/firecrown_developer_clp

TXPipe

# IN2P3
conda activate /sps/lsst/groups/clusters/cl_pipeline_project/conda_envs/txpipe_clp

# NERSC
conda activate /global/cfs/projectdirs/lsst/groups/CL/cl_pipeline_project/conda_envs/txpipe_clp

To reproduce the shared DESC environments locally instead:

Note: Do not use pip directly to install the required packages, as not all dependencies can be installed this way. Instead, use conda, then pip to install clpipe.

conda env update -f txpipe_environment.yml
conda activate txpipe_clp
pip install .
conda deactivate


conda env update -f firecrown_environment.yml
conda activate firecrown_clp
pip install .
conda env config vars set CSL_DIR=${CONDA_PREFIX}/cosmosis-standard-library
conda deactivate
conda activate firecrown_clp

cd ${CONDA_PREFIX}
source ${CONDA_PREFIX}/bin/cosmosis-configure
cosmosis-build-standard-library main

Switch between environments with:

conda activate firecrown_clp
# or
conda activate txpipe_clp

Pipeline Structure

This repository connects three DESC tools through ceci:

• TXPipe — data computation, produces a SACC file
• TJPCov — covariance matrix computation, updates the SACC file
• Firecrown — prepares the inputs required for MCMC inference

Running the pipeline requires two configuration files:

1. CL_concat.yml — the pipeline config. Defines which stages run, their inputs/outputs, parallelization, and other global settings (e.g. whether MPI is used).
2. config.yml — the stage config. Contains parameters specific to each stage; any modeling change should be made here.

TXPipe, TJPCov, and Firecrown each need their own stage configuration, since they produce different outputs and run in different Conda environments.

Pipeline flow:

TXPipe    → SACC file (data vectors)
TJPCov    → SACC file + covariance matrix
Firecrown → inference inputs

A separate job must be run to execute the final Firecrown inference — it is not launched automatically as part of the ceci pipeline.

Note: due to missing components in TJPCov, the CLPCovariance stage currently performs additional computations to rescale the covariance matrix.

Running a single stage

Every stage can also be run standalone, without a full pipeline file, since each PipelineStage subclass exposes its own CLI:

python -m clpipe <StageName> --config=<config.yml> --<tag>=<path> ...

This is mainly useful for debugging a single stage in isolation.

Example Runs

The examples directory contains sample pipeline executions:

• examples/cosmodc2_redmapper — the cosmoDC2 redMaPPer analyses
• examples/cosmoDC2-1deg2 — a full TXPipe to CosmoSIS run on the 1 deg² TXPipe example data
• examples/cosmodc2_halos — cluster counts from cosmoDC2 halos, a manually constructed pipeline, useful as a reference if you'd rather not use the template
• examples/capish_simulation — cluster counts and stacked ΔΣ profiles from the mean of capish mock realizations
• examples/template — scripts to generate the pipeline and stage configuration files (currently built for CosmoDC2; support for other analyses is under construction)

Each run directory has one ceci file per step it runs (TXPipe.yml, TJPCov.yml, Firecrown.yml), one stage configuration file and a job script that runs them in order, e.g.:

cd examples/cosmodc2_redmapper/baseline/cosmodc2_redmapper_full_analysis/run_in2p3_mor
sbatch launch_job_all_in2p3.sh

Downloading example data

Large chains, sacc files, and (for capish) mock realizations are not committed to git -- the notebooks under examples/cosmodc2_redmapper/ and examples/capish_simulation/ load them from CLPipe/data/, hosted on the NERSC Science Gateway portal:

https://portal.nersc.gov/cfs/lsst/clpipe/data/

Download the whole tree into the right place with:

wget -r -np -nH --cut-dirs=3 -R "index.html*" \
  https://portal.nersc.gov/cfs/lsst/clpipe/data/ \
  -P CLPipe/

This lands files at CLPipe/data/fiducial_cosmology_cosmodc2.yml, CLPipe/data/cosmodc2_redmapper/{chains,sacc}/ and CLPipe/data/capish_simulation/{chains,*.sacc,data_generation/mocks_seeds}/, matching the relative paths the notebooks expect. See data/README.md for what is hosted there and how to refresh it.

Testing

Tests live under tests/ and run via pytest, split into two tiers:

• Fast — only needs ceci installed. Checks file generation and CLI wiring, no real computation.

  pytest tests -m "not slow"

• Full — needs the full DESC stack (pyccl, tjpcov, crow, firecrown, cosmosis). Runs real covariance computation and cosmosis sampler runs.

  pytest tests

CI (.github/workflows/ci.yml) runs the fast tier first and only starts the full-stack tier once it passes.

Contributing

• Open a pull request against main; CI runs automatically (skipped for draft PRs).
• The fast tier (pytest tests -m "not slow") gates the full-stack tier — if your PR fails fast, the full-stack job never starts.
• Add or update tests under tests/ for any change to stage behavior, generated config files, or the Conda environments.
• For questions or to get involved with the Cluster Working Group, reach out to the maintainers listed in pyproject.toml.