Byte swapped CRTM binary files required

[ADCollard]

Describe the bug

I am trying to run GSI with spack-stack-2.0.0 on URSA. I am trying to run with the CRTM binary files (as I get seg faults when trying to use netCDF).

It appears that there is an inconsistency with endianness between what is expected from by spack and what is available in the fix directory:

 READ_IASI:  iasi offset         2554
 READ_IASI: crtm_spccoeff_load() on path "./crtm_coeffs/"
 Check_Binary_File(FAILURE) : Data file needs to be byte-swapped.
 Check_Binary_File(FAILURE) : Data file needs to be byte-swapped.
 Check_Binary_File(FAILURE) : Data file needs to be byte-swapped.
 Check_Binary_File(FAILURE) : Data file needs to be byte-swapped.
 Open_Binary_File(FAILURE) : Error checking ./crtm_coeffs/iasi_metop-c.SpcCoeff.bin file byte order

The fix directory on Ursa is /contrib/spack-stack/spack-stack-2.0.0/envs/ue-oneapi-2025.2.1/install/none/none/crtm-fix-3.1.2.0-ipi7gdt/fix

This appears to be related to #1561.

To Reproduce
I can provide scripts on request.

Expected behavior
The CRTM should be able to read the binary fix files without issue.

System:
Noticed on Ursa, but probably on other systems.

[climbfuji]

Can you clarify what you mean with "what is expected from by spack" and "The CRTM should be able to read the binary fix files without issue."

I don't know exactly which code is throwing the error, but it looks to me that the error is in the GSI code?

I am pretty sure that @BenjaminTJohnson tested the crtm-fix-3.1.2.0 tarball with crtm@3.1.2 and that there is no byte-swap issue. @BenjaminTJohnson Can you comment, please?

[ADCollard]

Can you clarify what you mean with "what is expected from by spack" and "The CRTM should be able to read the binary fix files without issue."

I don't know exactly which code is throwing the error, but it looks to me that the error is in the GSI code?

I am pretty sure that @BenjaminTJohnson tested the crtm-fix-3.1.2.0 tarball with crtm@3.1.2 and that there is no byte-swap issue. @BenjaminTJohnson Can you comment, please?

@climbfuji Thanks for your reply. I would be happy to learn that this is just a GSI issue. And any guidance on how to test this independently of GSI would be appreciated.

All I have done is pointed GSI at spack-stack-2.0.0 via gsi_ursa.intel.lua:

prepend_path("MODULEPATH", "/contrib/spack-stack/spack-stack-2.0.0/envs/ue-oneapi-2025.2.1/modules/Core")

local stack_oneapi_ver=os.getenv("stack_oneapi_ver") or "2025.2.1"
local stack_impi_ver=os.getenv("stack_impi_ver") or "2021.13"
local oneapi_mkl_ver=os.getenv("oneapi_mkl_ver") or "2025.2.0"
local cmake_ver=os.getenv("cmake_ver") or "3.31.8"
local crtm_fix_ver=os.getenv("crtm_fix_ver") or "3.1.2"

load(pathJoin("stack-intel-oneapi-compilers", stack_oneapi_ver))
load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver))
load(pathJoin("intel-oneapi-mkl", oneapi_mkl_ver))
load(pathJoin("cmake", cmake_ver))

In the existing GSI configuration we point to:

prepend_path("MODULEPATH", "/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.2.1/install/modulefiles/Core")

local stack_oneapi_ver=os.getenv("stack_oneapi_ver") or "2024.2.1"
local stack_impi_ver=os.getenv("stack_impi_ver") or "2021.13"
local oneapi_mkl_ver=os.getenv("oneapi_mkl_ver") or "2024.2.1"
local stack_python_ver=os.getenv("stack_python_ver") or "3.11.7"
local cmake_ver=os.getenv("cmake_ver") or "3.27.9"
local crtm_fix_ver=os.getenv("crtm_fix_ver") or "2.4.0.2"

load(pathJoin("stack-oneapi", stack_oneapi_ver))
load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver))
load(pathJoin("intel-oneapi-mkl", oneapi_mkl_ver))
load(pathJoin("stack-python", stack_python_ver))
load(pathJoin("cmake", cmake_ver))

The latter will read in the coefficient files just fine, whereas the former gives the byte-swap error.

I naively thought that maybe the issue is related to how the CRTM is being compiled on Ursa for spack-stack-2.0.0 vs spack-stack-1.9.2.

[climbfuji]

What I need to understand is which code (maybe you can point me to the line of code in GitHub?) throws the error

Open_Binary_File(FAILURE) : Error checking ./crtm_coeffs/iasi_metop-c.SpcCoeff.bin file byte order

Is that code in CRTM or in GSI?

[ADCollard]

It's in CRTM:
https://github.com/JCSDA/CRTMv3/blob/develop/src/Utility/Binary_File_Utility.f90#L205

BTW, the coefficient files stored in /contrib/spack-stack/spack-stack-2.0.0/envs/ue-oneapi-2025.2.1/install/none/none/crtm-fix-3.1.2.0-ipi7gdt/fix (which I am trying to read) are big endian.

[ADCollard]

@climbfuji I have confirmed that is all runs fine when I use Little Endian files (obtained directly from CRTM).

Using little endian is fine as far as I am concerned and we can always use our own versions. I am just not sure how the coefficient files that come with Spack-Stack are managed.

[BenjaminTJohnson]

export F_UFMTENDIAN=little

This will always ensure that it uses LE.

[climbfuji]

@ADCollard I am of course still confused, but if your issue is that the files are Big Endian whereas you expect them in Little Endian, then maybe setting F_UFMTENDIAN=big will solve it? You can do that for individual iounits, too. See

https://sepwww.stanford.edu/sep/claudio/Research/Prst_ExpRefl/ShtPSPI/intel/fce/10.1.015/doc/main_for/mergedProjects/bldaps_for/common/bldaps_envarufm.htm

Now, looking at what Spack does: https://github.com/JCSDA/spack-packages/blob/64cccad58b7806c458e8d86d7fb9d864c025e82a/repos/spack_repo/builtin/packages/crtm_fix/package.py

NOAA-EMC (!) back in the days put in the logic to install either that big or little endian files directory in the top-level fix directory, rather than keeping the untarred directory structure (which has subdirectories for big endian and little endian) in place. This is the commit from Kyle Gerheiser from 4 years ago: https://github.com/JCSDA/spack/blame/dde1a3235f6eb7a4388bb1a5325f93a70d49f63d/var/spack/repos/builtin/packages/crtm-fix/package.py

I don't know why he chose big endian as the default back then. Note that there is an open issue #1910 that also has to do with the fact that these directory structures are flattened when they shouldn't be.

In short, if EMC were to give up on the requirement of a flat directory structure and instead used the extracted tarball as is, then not only your but also my issues would be solved!

@AlexanderRichert-NOAA FYI

[ADCollard]

Thanks @climbfuji and @BenjaminTJohnson.

Currently we have a separate repository for the exact layout of coefficients that we need for the GSI. So if you want to move the spack-stack CRTM fix to the original structure then I think that would be OK from my perspective (but I am sure others would have opinions)

The flat structure has been imposed by NCO and I do not know if that is likely to change or not.

Anyway, I think I have a way forward for now and hopefully I can get netCDF to work so all this is moot!

Thanks.

[climbfuji]

Thanks @climbfuji and @BenjaminTJohnson.

Currently we have a separate repository for the exact layout of coefficients that we need for the GSI. So if you want to move the spack-stack CRTM fix to the original structure then I think that would be OK from my perspective (but I am sure others would have opinions)

The flat structure has been imposed by NCO and I do not know if that is likely to change or not.

Anyway, I think I have a way forward for now and hopefully I can get netCDF to work so all this is moot!

Thanks.

Moving to netCDF sounds like the best solution, I agree. Nevertheless, if NCO wants a flat directory structure and doesn't care what the developer of the package think is the best solution, then we can do that with symbolic links (instead of moving files); but we'll need to link the correct files (little endian I suppose for NCO as well instead of big endian)?