From c4de6bcc9cce1b66916e91d0a3930db32f6e5831 Mon Sep 17 00:00:00 2001 From: Amber Lim Date: Fri, 5 Sep 2025 17:21:34 -0500 Subject: [PATCH 1/3] Create staging directory transition page --- _uw-research-computing/staging-transition.md | 61 ++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 _uw-research-computing/staging-transition.md diff --git a/_uw-research-computing/staging-transition.md b/_uw-research-computing/staging-transition.md new file mode 100644 index 00000000..98f82e4a --- /dev/null +++ b/_uw-research-computing/staging-transition.md @@ -0,0 +1,61 @@ +--- +highlighter: none +layout: guide +title: "Staging directory transition" +guide: + category: General + tag: + - htc +--- + +## Announcement + +This fall, we are transitioning to a new directory structure for personal staging directories. **This affects all users on the HTC system**. + +Personal staging directories will now be located in **alphabetized subdirectories** based on the first letter of your NetID. For example: + + +| Previous `/staging` directory path | New `/staging` directory path | +| --- | --- | +| `/staging/netid` | `/staging/n/netid` | +| `/staging/bucky` | `/staging/b/bucky` | + +Group `/staging` directories are not affected, and will remain in the `/staging/groups` subdirectory. + +## Transition timeline and process + +We will start transitioning users' personal `/staging` directories to this new structure **starting Thursday, September 11, 2025**. + +CHTC will: + +1. Copy your files to your new `/staging` directory. +2. Create a symlink at your previous `/staging` directory path that points to the new `/staging` directory path. + +We will complete this transition on **Wednesday, December 31** and delete all symlinks. + +### What you should do + +If you have an existing `/staging` directory, **between September 11 and December 31**, please review all your files that reference your `/staging` directory. This may include but are not limited to: + +* HTCondor submit files +* Executables and scripts +* Environment variables +* DAGMan files + +Please change any reference to your personal `/staging` directory to the new path. + +For example, in an HTCondor submit file, change: + +``` +container_image = osdf:///chtc/staging/netid/my-container.sif +``` + +to: + +``` +container_image = osdf:///chtc/staging/n/netid/my-container.sif +``` + +## Get support + +If you have questions or concerns, please email us at [chtc@cs.wisc.edu](mailto:chtc@cs.wisc.edu). \ No newline at end of file From b13f4f0d017a3cf922f4e4b3dc1347aba823025d Mon Sep 17 00:00:00 2001 From: Amber Lim Date: Mon, 8 Sep 2025 13:01:36 -0500 Subject: [PATCH 2/3] Create staging transition page --- _uw-research-computing/staging-transition.md | 30 +++++++++++++------- images/staging-transition.svg | 1 + 2 files changed, 21 insertions(+), 10 deletions(-) create mode 100644 images/staging-transition.svg diff --git a/_uw-research-computing/staging-transition.md b/_uw-research-computing/staging-transition.md index 98f82e4a..d53eae21 100644 --- a/_uw-research-computing/staging-transition.md +++ b/_uw-research-computing/staging-transition.md @@ -8,9 +8,14 @@ guide: - htc --- -## Announcement -This fall, we are transitioning to a new directory structure for personal staging directories. **This affects all users on the HTC system**. +

Illustration of the `/staging` directory structure transition. On the left hand panel, the `/staging` directories has the subdirectories `alice`, `alison`, `bucky`, and `cathy`. An arrow points to the right hand panel, which has the `/staging` directory with the subdirectories `a`, `b`, and `c`. Subdirectories `alice` and `alison` are in the `a` subdirectory, `bucky` in the `b` subdirectory, and `cathy` in the `c` subdirectory.

+

Illustration of the /staging directory structure transition.

+ + +## Transition to a new `/staging` directory structure + +Starting Thursday, September 11, 2025, we are transitioning to a new directory structure for personal staging directories. **This affects all users on the HTC system**. Personal staging directories will now be located in **alphabetized subdirectories** based on the first letter of your NetID. For example: @@ -20,22 +25,23 @@ Personal staging directories will now be located in **alphabetized subdirectorie | `/staging/netid` | `/staging/n/netid` | | `/staging/bucky` | `/staging/b/bucky` | -Group `/staging` directories are not affected, and will remain in the `/staging/groups` subdirectory. - -## Transition timeline and process +Group `/staging` directories are not affected and will remain in the `/staging/groups` subdirectory. -We will start transitioning users' personal `/staging` directories to this new structure **starting Thursday, September 11, 2025**. +## Transition process CHTC will: 1. Copy your files to your new `/staging` directory. -2. Create a symlink at your previous `/staging` directory path that points to the new `/staging` directory path. +2. Create a [symlink](https://en.wikipedia.org/wiki/Symbolic_link) at your previous `/staging` directory path that points to the new `/staging` directory path. -We will complete this transition on **Wednesday, December 31** and delete all symlinks. +## Timeline -### What you should do +* **September 11, 2025**. Transition begins. Users may begin using their new `/staging` directory. +* **December 31, 2025**. Transition ends. Symlinks at previous `/staging` directory paths will be deleted. -If you have an existing `/staging` directory, **between September 11 and December 31**, please review all your files that reference your `/staging` directory. This may include but are not limited to: +## What you should do + +If you have an existing `/staging` directory, **between September 11 and December 31**, please review all your files that reference your `/staging` directory. This may include but is not limited to: * HTCondor submit files * Executables and scripts @@ -56,6 +62,10 @@ to: container_image = osdf:///chtc/staging/n/netid/my-container.sif ``` +## Why we are transitioning + +Our `/staging` directories are backed by the Ceph File System, which has [slower performance when it must load very large directories](https://docs.ceph.com/en/reef/cephfs/app-best-practices/#very-large-directories). To mitigate large loads on the file system, we are sorting users personal `/staging` directories into alphabetized subdirectories. + ## Get support If you have questions or concerns, please email us at [chtc@cs.wisc.edu](mailto:chtc@cs.wisc.edu). \ No newline at end of file diff --git a/images/staging-transition.svg b/images/staging-transition.svg new file mode 100644 index 00000000..5035d948 --- /dev/null +++ b/images/staging-transition.svg @@ -0,0 +1 @@ +/staging|--alice|--alison|--bucky`--cathy/staging|--a|   |--alice|   `--alison|--b|   `--bucky`--c`--cathy \ No newline at end of file From 150d659cb2a1fda60e4bf341ac498fa3c90f73c2 Mon Sep 17 00:00:00 2001 From: Amber Lim Date: Tue, 9 Sep 2025 16:07:05 -0500 Subject: [PATCH 3/3] Change staging examples to new directory structure --- _uw-research-computing/apptainer-htc.md | 4 +-- _uw-research-computing/check-quota.md | 14 +++++----- .../file-avail-largedata.md | 28 +++++++++++-------- _uw-research-computing/high-memory-jobs.md | 9 ++---- .../htc-docker-to-apptainer.md | 2 +- .../htc-job-file-transfer.md | 19 ++++--------- _uw-research-computing/htc-overview.md | 8 +++--- .../htc-uwdf-researchdrive.md | 2 +- 8 files changed, 41 insertions(+), 45 deletions(-) diff --git a/_uw-research-computing/apptainer-htc.md b/_uw-research-computing/apptainer-htc.md index 2061e268..d63795e2 100644 --- a/_uw-research-computing/apptainer-htc.md +++ b/_uw-research-computing/apptainer-htc.md @@ -138,7 +138,7 @@ exit Once you are satisfied that your container is built correctly, copy your `.sif` file to your staging directory. ``` -mv my-container.sif /staging/$USER +mv my-container.sif /staging/u/username ``` {:.term} @@ -314,7 +314,7 @@ Since Apptainer `.sif` files are routinely more than 1GB in size, we recommend t It is usually easiest to move the container file directly to staging while still in the interactive build job: ``` -mv my-container.sif /staging/$USER +mv my-container.sif /staging/u/username ``` {:.term} diff --git a/_uw-research-computing/check-quota.md b/_uw-research-computing/check-quota.md index f44bd614..735a03b1 100644 --- a/_uw-research-computing/check-quota.md +++ b/_uw-research-computing/check-quota.md @@ -53,9 +53,9 @@ This will print a table with your `/home` and `/staging` quotas. An example outp ``` [user@ap2002 ~]$ get_quotas -Path Disk_Used(GB) Disk_Limit(GB) Files_Used File_Limit -/home/user 16.0711 40 8039 N/A -/staging/user 13.4731 100 12 1000 +Path Disk_Used(GB) Disk_Limit(GB) Files_Used File_Limit +/home/username 16.0711 40 8039 N/A +/staging/u/username 13.4731 100 12 1000 ``` {:.term} @@ -119,10 +119,10 @@ _____________________________________________________________________ == NOTICE: THIS NODE IS ON PUPPET ENVIRONMENT "puppet8" == Filesystem quota report (last updated 10:33 AM) -Storage Used (GB) Limit (GB) Files (#) File Cap (#) Quota (%) ------------------- ----------- ------------ ----------- -------------- ----------- -/home/user 29.38 40 94 0 73.46 -/staging/user 50.23 1000 110 10000 5.02 +Storage Used (GB) Limit (GB) Files (#) File Cap (#) Quota (%) +------------------ ----------- ------------ ----------- -------------- ----------- +/home/username 29.38 40 94 0 73.46 +/staging/u/username 50.23 1000 110 10000 5.02 ``` {:.term} diff --git a/_uw-research-computing/file-avail-largedata.md b/_uw-research-computing/file-avail-largedata.md index 54f0e87a..1c59311c 100644 --- a/_uw-research-computing/file-avail-largedata.md +++ b/_uw-research-computing/file-avail-largedata.md @@ -26,7 +26,7 @@ When submitting jobs to the HTC system, large data needs to be stored and handle * [Intended use](#intended-use) * [User responsibilities](#user-responsibilities) - [Stage large data](#stage-large-data) - * [Request a `/staging` directory](#request-a-staging-directory) + * [Your personal `/staging` directory](#your-personal-staging-directory) * [Reduce file counts](#reduce-file-counts) * [Use the transfer server](#use-the-transfer-server) * [Remove files after jobs complete](#remove-files-after-jobs-complete) @@ -67,18 +67,24 @@ CHTC staff reserve the right to remove data from our large data staging location In order to stage large data for use on CHTC's HTC system, users must: -1. **Request a `/staging` directory**: Use our quota request form. 1. **Reduce file counts**: Combine and compress files that are used together. 1. **Transfer files to the HTC system via the transfer server**: Upload your data via our dedicated file transfer server. 1. **Remove files after jobs complete**: Our data staging space is quota-controlled and not backed up. -### Request a `/staging` directory +### Your personal `/staging` directory -Any one with a CHTC account whose data meets the intended use above can request space in our large data staging area by filling out a quota request form. The default quota is 100 GB / 1000 items; if a larger quota is needed, request a higher quota. The created directory will exist at this path: `/staging/username` +Each user should have a personal `/staging` directory. The created directory will exist in an alphabetized subdirectory based on the **first letter** of your NetID. For example: + +| NetID | Path to your personal `/staging` directory | +| --- | --- | +| `alice` | `/staging/a/alice` | +| `bucky` | `/staging/b/bucky` | We can also create group or shared spaces by request. -

Need a /staging directory or higher quota?

+The default quota is 100 GB / 1000 items; if a larger quota is needed, request a higher quota. + +

Need a group /staging directory or higher quota?

Quota request form @@ -88,7 +94,7 @@ We can also create group or shared spaces by request. ### Reduce file counts -The file system backing our `/staging`space is optimized to handle small numbers of large files. If your job requires many small files, we recommend placing these files in the `/home` directory or compressing multiple files into a single zip file or tarball. See [this table](htc-job-file-transfer#data-storage-locations) for more information on the differences between `/staging` and `/home`. +The file system backing our `/staging` space is optimized to handle small numbers of large files. If your job requires many small files, we recommend placing these files in the `/home` directory or compressing multiple files into a single zip file or tarball. See [this table](htc-job-file-transfer#data-storage-locations) for more information on the differences between `/staging` and `/home`. Data placed in our large data `/staging` location should be stored in as few files as possible (ideally, one file per job), and will be used by a job only after being copied from `/staging` into the job working directory. Similarly, large output should first be written to the job's working directory then compressed in to a single file before being copied to `/staging` at the end of the job. @@ -105,7 +111,7 @@ Uploading or downloading data to `/staging` should only be performed via CHTC's For example, you can use `scp` to transfer files into your `/staging` directory: ``` -$ scp large.file netid@transfer.chtc.wisc.edu:/staging/netid/ +$ scp large.file username@transfer.chtc.wisc.edu:/staging/u/username/ ``` {:.term} @@ -123,7 +129,7 @@ Staged files should be specified in the job submit file using the `osdf:///` or depending on the size of the files to be transferred. [See this table for more information](htc-job-file-transfer#transfer-input-data-to-jobs-with-transfer_input_files). ``` -transfer_input_files = osdf:///chtc/staging/username/file1, file:///staging/username/file2, file3 +transfer_input_files = osdf:///chtc/staging/u/username/file1, file:///staging/u/username/file2, file3 ``` {:.sub} @@ -136,7 +142,7 @@ Large outputs should be transferred to staging using the same file transfer prot ``` transfer_output_files = file1, file2, file3 -transfer_output_remaps = "file1 = osdf:///chtc/staging/username/file1; file2 = file:///staging/username/file2" +transfer_output_remaps = "file1 = osdf:///chtc/staging/u/username/file1; file2 = file:///staging/u/username/file2" ``` {:.sub} @@ -166,14 +172,14 @@ within the user's `/home` directory: ``` {.sub} ### Example submit file for a single job that stages large data # Files for the below lines MUST all be somewhere within /home/username, -# and not within /staging/username +# and not within /staging/u/username executable = run_myprogram.sh log = myprogram.log output = $(Cluster).out error = $(Cluster).err -transfer_input_files = osdf:///chtc/staging/username/myprogram, file:///staging/username/largedata.tar.gz +transfer_input_files = osdf:///chtc/staging/u/username/myprogram, file:///staging/u/username/largedata.tar.gz # IMPORTANT! Require execute servers that can access /staging Requirements = (Target.HasCHTCStaging == true) diff --git a/_uw-research-computing/high-memory-jobs.md b/_uw-research-computing/high-memory-jobs.md index 0242a2dc..a8cf37f9 100644 --- a/_uw-research-computing/high-memory-jobs.md +++ b/_uw-research-computing/high-memory-jobs.md @@ -174,16 +174,13 @@ Altogether, a sample submit file may look something like this: ``` {.sub} ### Example submit file for a single staging-dependent job -universe = vanilla - # Files for the below lines will all be somewhere within /home/username, -# and not within /staging/username +# and not within /staging/u/username log = run_myprogram.log executable = run_Trinity.sh output = $(Cluster).out error = $(Cluster).err transfer_input_files = trinityrnaseq-2.0.1.tar.gz -should_transfer_files = YES # Require execute servers that have large data staging Requirements = (Target.HasCHTCStaging == true) @@ -236,7 +233,7 @@ Altogether, a sample script may look something like this (perhaps called #!/bin/bash # Copy input data from /staging to the present directory of the job # and un-tar/un-zip them. -cp /staging/username/reads.tar.gz ./ +cp /staging/u/username/reads.tar.gz ./ tar -xzvf reads.tar.gz rm reads.tar.gz @@ -255,7 +252,7 @@ Trinity --seqType fq --left reads_1.fq \ # Trinity will write output to the working directory by default, # so when the job finishes, it needs to be moved back to /staging tar -czvf trinity_out_dir.tar.gz trinity_out_dir -cp trinity_out_dir.tar.gz trinity_stdout.txt /staging/username/ +cp trinity_out_dir.tar.gz trinity_stdout.txt /staging/u/username/ rm reads_*.fq trinity_out_dir.tar.gz trinity_stdout.txt ### END diff --git a/_uw-research-computing/htc-docker-to-apptainer.md b/_uw-research-computing/htc-docker-to-apptainer.md index a8847f26..bbcb31b3 100644 --- a/_uw-research-computing/htc-docker-to-apptainer.md +++ b/_uw-research-computing/htc-docker-to-apptainer.md @@ -78,7 +78,7 @@ INFO: Build complete: container.sif Because container images are generally large, we require users to move these images into their staging directories. While you are still in your interactive job, move the image to your staging directory. ``` -mv container.sif /staging/username/ +mv container.sif /staging/u/username/ ``` {:.term} diff --git a/_uw-research-computing/htc-job-file-transfer.md b/_uw-research-computing/htc-job-file-transfer.md index 8659664d..6ce81c4a 100644 --- a/_uw-research-computing/htc-job-file-transfer.md +++ b/_uw-research-computing/htc-job-file-transfer.md @@ -42,13 +42,6 @@ The HTC system has two primary locations where users can place their files: The data management mechanisms behind `/home` and `/staging` are different and are optimized to handle different file sizes and numbers of files. It's important to place your files in the correct location to improve the efficiency at which your data is handled and maintain the stability of the HTC file systems. -

Need a /staging directory?

-
-
- Request one here -
-
- ## Transfer input data to jobs with `transfer_input_files` @@ -57,8 +50,8 @@ To transfer files to jobs, we must specify these files with `transfer_input_file | Input File Size (Per File)* | File Location | Submit File Syntax to Transfer to Jobs | | ----------- | ----------- | ----------- | ----------- | | 0 - 1 GB | `/home` | `transfer_input_files = input.txt` | -| 1 - 30 GB | `/staging` | `transfer_input_files = osdf:///chtc/staging/NetID/input.txt` | -| 30 - 100 GB | `/staging` | `transfer_input_files = file:///staging/NetID/input.txt` | +| 1 - 30 GB | `/staging` | `transfer_input_files = osdf:///chtc/staging/u/username/input.txt` | +| 30 - 100 GB | `/staging` | `transfer_input_files = file:///staging/u/username/input.txt` | | 1 - 100 GB | `/staging/groups`† | `transfer_input_files = file:///staging/groups/group_dir/input.txt` | | 100 GB+ | | Contact the facilitation team about the best strategy to stage your data | @@ -73,7 +66,7 @@ Multiple input files and file transfer protocols can be specified and delimited ``` # My job submit file -transfer_input_files = file1, osdf:///chtc/staging/username/file2, file:///staging/username/file3, dir1, dir2/ +transfer_input_files = file1, osdf:///chtc/staging/u/username/file2, file:///staging/u/username/file3, dir1, dir2/ requirements = (HasCHTCStaging == true) @@ -120,7 +113,7 @@ transfer_output_files = output_file, output/output_file2, output/output_file3 To transfer files back to `/staging` or a specific directory in `/home`, you will need an additional line in your HTCondor submit file, with each item separated by a semicolon (;): ``` -transfer_output_remaps = "output_file = osdf:///chtc/staging/NetID/output1.txt; output_file2 = /home/netid/outputs/output_file2" +transfer_output_remaps = "output_file = osdf:///chtc/staging/u/username/output1.txt; output_file2 = /home/u/username/outputs/output_file2" ``` {:.sub} @@ -133,7 +126,7 @@ Make sure to only include one set of quotation marks that wraps around the infor If you want to transfer *all* files to a specific destination, use `output_destination`: ``` -output_destination = osdf:///chtc/staging/netid/ +output_destination = osdf:///chtc/staging/u/username/ ``` {:.sub} @@ -146,7 +139,7 @@ The `osdf:///` file transfer plugin is powered by the [Pelican Platform](https:/ To transfer and unpack files, append a `?pack=auto` at the end of the plugin path of the compressed object to be transferred. ``` -transfer_input_files = osdf:///chtc/staging/netid/filename.tar.gz?pack=auto, input1.txt, input2.txt +transfer_input_files = osdf:///chtc/staging/u/username/filename.tar.gz?pack=auto, input1.txt, input2.txt ``` This feature is only availble for Pelican-based plugins (`osdf://`, `pelican://`) and is not available for `file://` or normal file transfers. This feature is also not recommended for compressed files larger than 30 GB. diff --git a/_uw-research-computing/htc-overview.md b/_uw-research-computing/htc-overview.md index c92dc353..d752e99f 100644 --- a/_uw-research-computing/htc-overview.md +++ b/_uw-research-computing/htc-overview.md @@ -163,9 +163,9 @@ Each of the disk space values are given in megabytes (MB), which can be converte ### Check `/staging` Quota and Usage -To see your `/staging` quota and usage, use the `get_quotas ` command. For example, +To see your `/staging` quota and usage, use the `get_quotas` command. For example, ``` -[NetID@ap2001 ~]$ get_quotas /staging/NetID +[NetID@ap2001 ~]$ get_quotas ``` {:.term} @@ -178,8 +178,8 @@ Alternatively, the `ncdu` command can also be used to see how many files and directories are contained in a given path: ``` -[NetID@ap2001 ~]$ ncdu /home/NetID -[NetID@ap2001 ~]$ ncdu /staging/NetID +[NetID@ap2001 ~]$ ncdu /home/username +[NetID@ap2001 ~]$ ncdu /staging/u/username ``` {:.term} diff --git a/_uw-research-computing/htc-uwdf-researchdrive.md b/_uw-research-computing/htc-uwdf-researchdrive.md index 7ef5360e..323c558e 100644 --- a/_uw-research-computing/htc-uwdf-researchdrive.md +++ b/_uw-research-computing/htc-uwdf-researchdrive.md @@ -84,7 +84,7 @@ transfer_output_files = outputfile1.txt, outputfile2.txt, outputfile3.txt You can use `transfer_output_remaps` to place files in different locations: ``` -transfer_output_remaps = "outputfile1.txt = pelican://chtc.wisc.edu/researchdrive//CHTC/outputfile1.txt; outputfile2.txt = osdf:///chtc/staging//outputfile2.txt" +transfer_output_remaps = "outputfile1.txt = pelican://chtc.wisc.edu/researchdrive//CHTC/outputfile1.txt; outputfile2.txt = osdf:///chtc/staging/u/username/outputfile2.txt" ``` The example above remaps the output files such that only `outputfile1.txt` is placed in ResearchDrive, `outputfile2.txt` is placed in `/staging`, and `outputfile3.txt` is placed in the submit directory on `/home`.