Apptainer build temp directory cleanup + best practice question

[dhjoo98]

Hi,

I’m working on building Apptainer container images for a GPU research project (Omniprobe) on the mi300x partition, and I wanted to report an issue related to Apptainer temporary directories and filesystem limits. I also wanted to ask for guidance on the recommended build workflow for large container images on this cluster.

⸻

Background

By default, Apptainer uses /tmp during image extraction/build. On the compute nodes, /tmp is backed by a 50G local disk (/dev/vdb), which is insufficient for our container image.

Our workflow builds from a Bootstrap: localimage SIF containing:

• full LLVM build
• Triton
• PyTorch
• additional GPU research dependencies

During extraction, the sandbox exceeds 50G and fails with:

No space left on device

To work around this, we redirected Apptainer temporary/cache storage to /work1:

export APPTAINER_TMPDIR=/work1/baharasgari/djoo/apptainer_tmp
export APPTAINER_CACHEDIR=/work1/baharasgari/djoo/apptainer_cache

⸻

Issue Encountered on /work1

When building on /work1, extraction fails very early (typically after ~84–135 files) with:

FATAL ERROR: write_file: failed to create file ..., because Too many open files in system

Additional notes:

• This occurs even when limiting unsquashfs to a single processor:

taskset -c 0-0

• /proc/sys/fs/file-nr indicates that the global kernel file limit is effectively unlimited.
• Based on the behavior, we suspect this may instead be related to a Lustre per-session/per-client open-file limit rather than a Linux kernel-wide limit.

⸻

Current Problem

One of the compute nodes (k002-004-v8) went down during a build, leaving behind partially extracted sandbox directories under:

/work1/baharasgari/djoo/apptainer_tmp/

Now, attempts to clean these directories from the login node fail with the same open-file-related errors:

rm: cannot remove '...build-temp-.../rootfs': Too many open files in system
find: '/work1/baharasgari/djoo/apptainer_tmp': Too many open files in system

The leftover directories appear to be:

/work1/baharasgari/djoo/apptainer_tmp/build-temp-1997835604/
/work1/baharasgari/djoo/apptainer_tmp/build-temp-2676158589/
/work1/baharasgari/djoo/apptainer_tmp/build-temp-3480337836/

⸻

Questions

1. What is the recommended way to safely remove these leftover directories given the open-file errors?
   • Is this something administrators need to clean up?
   • Or is there a recommended user-side cleanup procedure?
2. Is there a better-supported location or workflow for large Apptainer builds on this cluster?
   • For example, a dedicated build partition
   • Larger node-local scratch storage
   • Recommended temporary filesystem configuration
3. Is there a Lustre per-session/per-client open-file limit associated with these failures?
   • If so, is it possible to request a higher limit for container build workloads?

Thank you for your help.