NOMAI 2.0: REGALADE host photo-z, lower duration cutoff, Rainbow wavelength fix, new feature

[erusseil]

Linked to issue(s): #715

If this is a new release, did you issue the corresponding schema in fink-client?

No schema change: superluminous_score's output type/contract is unchanged.

What changes were proposed in this pull request?

This is the full "NOMAI 1.0 -> 2.0" rework of the superluminous supernovae
(SLSN) classifier, covering 10 commits:

• Lower the age cutoff from 30 to 20 days (kernel.min_duration), so
  the classifier scores sources earlier in their evolution.
• Fix the Rainbow fit's ZTF effective wavelengths
  (kernel.band_wave_aa): {1: 4770.0, 2: 6231.0} ->
  {1: 4746.48, 2: 6366.38} (correct SVO Filter Profile Service values for
  the g/r bands).
• New ntrends feature (slsn_classifier.ntrend_changes): counts
  abrupt trend reversals in the light curve (a real SLSN should rise/decay
  smoothly), meant to flag noisy/bogus-looking light curves.
  -Extra post-hoc cuts Using n_trends and duration. These cuts were already used inside fink-broker. They are not directly merged in the main pipeline
• New brightness persistence and shape_irregularity features.
  Simple statistical features, cheap to compute and very informative to find SLSNe.
  Discovered with symbolic regression.
• New shape_irregularity feature : a simple statistical feature, cheap to compute
  very informative to distinguish SLSNe. Discovered through symbolic regression.
• New ntrends feature (slsn_classifier.ntrend_changes): counts
  abrupt trend reversals in the light curve (a real SLSN should rise/decay
  smoothly), meant to flag noisy/bogus-looking light curves.
• Replace the SDSS host photo-z with a REGALADE crossmatch (see below)
  -- the main change in this PR.
• Correct light curves for Milky Way extinction instead of using ebv
  as a raw feature: previously ebv was passed directly to the
  classifier, hoping it would implicitly learn extinction effects from it.
  New slsn_classifier.deredden_lightcurve corrects cflux/csigflux/
  cmagpsf for Milky Way extinction, band by band, before Rainbow/salt
  fitting and statistical features, so temperature, color and peak
  magnitude features are computed on the intrinsic light curve directly --
  no feature needed, and light curves become directly comparable across
  different lines of sight. extract_features no longer outputs an ebv
  column. abs_peak's signature is left untouched, since it's called
  elsewhere (e.g. fink-broker/bin/ztf/archive_slsn_candidates.py) with a
  real ebv value on non-dereddened magnitudes; processor.py now passes
  ebv=0 since peak_mag_g/peak_mag_r are already corrected by the
  time they reach it.
• abs_peak optimization: cosmo.luminosity_distance() was being
  called once per band per redshift offset (up to 6 calls); it doesn't
  depend on band, so it's now computed once for the 3 redshift offsets
  and reused across bands. Purely a speed change.
• Various docstring/doctest improvements across kernel.py,
  processor.py and slsn_classifier.py, and two ruff-formatting-only
  commits.
• The trained model itself (superluminous_classifier.joblib ->
  NOMAI.joblib, kernel.classifier_path updated accordingly), retrained
  twice: once after the initial rework, and again after the Milky Way
  extinction fix above, on a fresh feature extraction.

REGALADE host photo-z (the main change)

The classifier uses a host galaxy photo-z to compute a physical upper
bound on absolute magnitude (slsn_classifier.abs_peak), and forces the
SLSN probability to 0 when a source can't plausibly be superluminous even
in the brightest case. This photo-z previously came from a live query to
the SDSS SkyServer (get_sdss_photoz). This PR replaces it with an
offline ellipse-based crossmatch against the REGALADE galaxy catalog
(Tranin et al., https://github.com/htranin/regalade), which has much wider
sky coverage and a more accurate host association method (directional
light radius ellipse matching) than a simple nearest-neighbor SDSS query.

On our own labeled SLSN candidate sample, REGALADE finds a host for 80.6%
of objects vs. 52.1% for SDSS, and among objects matched by both, ~20%
differ by more than 0.05 in redshift (some by close to 1, i.e. clearly a
different galaxy) -- so this isn't just a coverage improvement, it also
fixes host misassociations that were silently corrupting the physical cut
in both directions (wrongly suppressing true SLSNe and wrongly letting
non-SLSNe through).

Concretely:

• slsn_classifier.get_sdss_photoz/add_all_photoz no longer make a
  network call to SDSS. add_all_photoz keeps the same signature/contract
  (still adds photoz/photozerr columns), so processor.py's call site
  is unchanged.
• New slsn_classifier.get_regalade_photoz(ra, dec) crossmatches against
  the REGALADE catalog using ellipse_xmatch.crossmatch_ellipses (new
  dependency, declared in setup.py), with the exact same matching
  parameters (dlr_factor=1.25, nbins=20) used to build the training
  set, so inference-time photo-z is produced the same way as training-time
  photo-z. The catalog is lazily loaded and cached at module level so a
  Spark worker only pays the FITS read once, not once per micro-batch.
• New kernel.regalade_path/regalade_dlr_factor/regalade_nbins
  constants. See fink_science/data/catalogs/README.md for what the
  catalog file is and how it's derived.

How was this patch tested?

• All doctests in slsn_classifier.py, kernel.py and processor.py
  pass locally, including the new/changed ones (get_regalade_photoz,
  add_all_photoz, abs_peak, deredden_lightcurve, ntrend_changes,
  statistical_features), including the full Spark-based extract_features
  doctest.
• get_regalade_photoz verified to reproduce the training pipeline's
  photo-z values exactly (to floating point) on known objects.
• abs_peak's optimization verified against the pre-optimization
  implementation over 2000 random trials (max diff ~1e-13), including NaN
  propagation.
• ruff format --check / ruff check clean on every commit.
• The full training pipeline (feature extraction, REGALADE crossmatch,
  hyperparameter search, training) was re-run end to end after the Milky
  Way extinction fix, on a fresh feature extraction.
• Not yet run through the full Spark test suite in this environment
  (needs the REGALADE catalog file and ellipse-xmatch installed, see
  above)

[erusseil]

Update (2026-09-14): REGALADE now sourced from the alert, not a local catalog

The REGALADE integration described above has changed since this PR was opened. REGALADE was merged into fink-broker's own crossmatch pipeline (xmatch_cds, a live circular/nearest-neighbour match against VizieR, the same mechanism used for Simbad/Gaia/VSX/etc.) rather than the offline full-catalog approach this PR originally proposed. As a result:

get_regalade_photoz no longer reads a local REGALADE catalog file. It now refines the REGALADE candidate fink-broker's apply_all_xmatch already attached to the alert (regalade_ra, regalade_dec, R1, R2, PA, z, ezin), applying the same DLR-ellipse separation test as before, just against that single candidate instead of searching the full catalog.
No more local regalade_minimal_ZTF.fits dependency, and the ellipse-xmatch runtime dependency is dropped from setup.py.

NOMAI.joblib retrained on the new method (honest bootstrap CV F1: 0.7131, vs. 0.7202 under the old method — a ~1% relative difference).
Depends on fink-broker's REGALADE xmatch_cds block already being live, and ideally on fink-broker#1239 merging (20″ radius) for the coverage numbers above to hold; at the original 1.2″ radius

[JulienPeloton]

Thanks @erusseil -- code is clean :-)

For future reference:

• Add SLSN candidates in the test sample
• Add call to xmatch_cds before performing classifiation, to fully test the pipeline with real values.