csf_fci requires better memory management

[MatthewRHermes]

Moving this issue over from MatthewRHermes/mrh#48 since csf_fci lives here now.

In two csf_solver functions (csf_fci.csf.pspace here and csf_fci.csf.make_hdiag_csf here), the memory usage is problematic due to the two-step evaluation of Hamiltonian matrix elements in the CSF basis, which requires the (temporary) construction of arrays of size quadratic with respect to the number of corresponding determinants. Massively open-shell low-spin wave functions become impossible in the current implementation around (16e,16o) because the corresponding determinants are too numerous to store the block Hamiltonian in memory. The relevant arrays should be split into blocks handled sequentially, as is done for DFT quadrature. In the mean time, segfaults and calculations abruptly killed by cluster daemon processes are symptoms of unmanaged memory usage, and indicate a region of the code where a memory-checking step should be added (as of 2025/08/03 this hasn’t happened in a while).

[MatthewRHermes]

csf_fci.csfstring.get_spin_evecs here also raises a memory bottleneck. This array should also be split into blocks using a generator, LinearOperator, or a direct low-level implementation of the matrix-vector product.