Crash on skyCorrectFromSlit when using small regions

[KathleenLabrie]

This happens with DRAGONS v3.2 (probably happens with previous versions too).

reduce S20240410S0025_aperturesFound.fits -p aperture_growth=10 order=1 hsigma=2.5 function=chebyshev regions=[960:1000,1130:1170] interactive=True -r skyCorrectFromSlit

   PRIMITIVE: skyCorrectFromSlit
   -----------------------------
/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/astropy/modeling/fitting.py:763: RuntimeWarning: invalid value encountered in divide
  lhs /= scl
 ** On entry to DLASCLS parameter number  4 had an illegal value
 ** On entry to DLASCLS parameter number  4 had an illegal value
 ** On entry to DLASCLS parameter number  4 had an illegal value
 ** On entry to DLASCLS parameter number  4 had an illegal value
 ** On entry to DLASCLS parameter number  5 had an illegal value
 ** On entry to DLASCLS parameter number  4 had an illegal value
ERROR - Traceback (most recent call last):
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/recipe_system/reduction/coreReduce.py", line 240, in runr
ERROR -     primitive_as_recipe()
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/recipe_system/utils/decorators.py", line 352, in gn
ERROR -     ret_value = fn(pobj, adinputs=adinputs, **fnargs)
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/recipe_system/utils/decorators.py", line 269, in gn
ERROR -     ret_value = fn(pobj, **kwargs)
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/geminidr/core/primitives_spect.py", line 3032, in skyCorrectFromSlit
ERROR -     sky_model = fit_1D(sky, weights=sky_weights, **final_parms[idx][eidx],
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/gempy/library/fitting.py", line 179, in __init__
ERROR -     self._fit(image, weights=weights, plot=plot)
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/gempy/library/fitting.py", line 301, in _fit
ERROR -     fitted_models, fitted_mask = fitter(
ERROR -   File "/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/astropy/modeling/fitting.py", line 987, in __call__
ERROR -     fitted_model = self.fitter(model, x, y, z, weights=weights, **kwargs)
ERROR -   File "/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/astropy/modeling/fitting.py", line 300, in wrapper
ERROR -     return func(self, model, x, y, z=z, **kwargs)
ERROR -   File "/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/astropy/modeling/fitting.py", line 810, in __call__
ERROR -     t_coef, resids, rank, sval = np.linalg.lstsq(
ERROR -   File "/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 2326, in lstsq
ERROR -     x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj)
ERROR -   File "/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 124, in _raise_linalgerror_lstsq
ERROR -     raise LinAlgError("SVD did not converge in Linear Least Squares")
ERROR - numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares
ERROR - SVD did not converge in Linear Least Squares
ERROR - reduce caught an unhandled exception.
ERROR - 
ERROR - SVD did not converge in Linear Least Squares
ERROR - Traceback (most recent call last):
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/recipe_system/scripts/reduce", line 87, in main
ERROR -     r_reduce.runr()
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/recipe_system/reduction/coreReduce.py", line 240, in runr
ERROR -     primitive_as_recipe()
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/recipe_system/utils/decorators.py", line 352, in gn
ERROR -     ret_value = fn(pobj, adinputs=adinputs, **fnargs)
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/recipe_system/utils/decorators.py", line 269, in gn
ERROR -     ret_value = fn(pobj, **kwargs)
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/geminidr/core/primitives_spect.py", line 3032, in skyCorrectFromSlit
ERROR -     sky_model = fit_1D(sky, weights=sky_weights, **final_parms[idx][eidx],
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/gempy/library/fitting.py", line 179, in __init__
ERROR -     self._fit(image, weights=weights, plot=plot)
ERROR -   File "/Users/klabrie/data/GemKLRepo/activedragons/gempy/library/fitting.py", line 301, in _fit
ERROR -     fitted_models, fitted_mask = fitter(
ERROR -   File "/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/astropy/modeling/fitting.py", line 987, in __call__
ERROR -     fitted_model = self.fitter(model, x, y, z, weights=weights, **kwargs)
ERROR -   File "/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/astropy/modeling/fitting.py", line 300, in wrapper
ERROR -     return func(self, model, x, y, z=z, **kwargs)
ERROR -   File "/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/astropy/modeling/fitting.py", line 810, in __call__
ERROR -     t_coef, resids, rank, sval = np.linalg.lstsq(
ERROR -   File "/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 2326, in lstsq
ERROR -     x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj)
ERROR -   File "/Users/klabrie/condaenvs/susd3.10_20240130/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 124, in _raise_linalgerror_lstsq
ERROR -     raise LinAlgError("SVD did not converge in Linear Least Squares")
ERROR - numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares
ERROR - 
ERROR - Reduce instance aborted.

Removing the regions or making them much bigger (~10 x) avoids the crash. As can see from the screenshot, it's a very straight forward fit, no idea why it would fail.

There's a vertical area of bad pixels on the input image around x=400, maybe that's where the fit fails. I can't investigate with the interactive tool right now because the column selection slider does not seem to do anything anymore. (that's another issue, reported on Slack).

Data to reproduce (still proprietary, limited access): https://drive.google.com/drive/folders/1eCK2dk2UrzxN8B1yOmkuONCNfN2ktLpB?usp=sharing

[chris-simpson]

My first guess is that there's a column which has only one good pixel in it. The error doesn't arise when fitting a spline. The fitting is done by fit_1D() regardless of the type of fit, but the spline-fitter has explicit code to deal with cases where there are more degrees of freedom than parameters (hence the "Unmasking all data." messages). Since there's not a problem with other GMOS datasets (where there are chip-gap columns with zero good pixels), my guess is that this only happens with one good pixel. This would also explain why increasing the size of the regions works (additional good pixels are brought into that column).