The Data Transfer module provides functionality for efficiently transferring parquet files from source locations to destination locations with integrity verification. It operates in three phases:
- Filtering: The
DataTransferFilterclass identifies parquet files that need to be transferred and generates unique destination paths. - Scheduling: The
DataTransferScheduleCreationclass creates a comprehensive schedule of all file transfers by combining filter tables. - Execution: The
DataTransferRunnerclass performs the actual file transfers with parallel processing using MPI, including MD5 hash verification.
The module handles two types of files:
- Success files (
successes.parquet): Contains data about successfully downloaded images - Error files (
errors.parquet): Contains data about download errors
Each file is transferred to a structured destination path that includes the server name and a randomly generated UUID to ensure uniqueness.
Add these values to your config file:
dst_image_folder: The destination folder where the image (successes) files will be copieddst_error_folder: The destination folder where the error files will be copied
Purpose: Executes the actual tool processing using MPI parallelism.
Key Components:
- Runs on the configured number of nodes with specified worker distribution
- Calls
main/runner.pywith the specified tool name - Reads the schedule created by the scheduler and processes assigned partitions
- Uses all allocated nodes for maximum parallelism
- Configures memory settings for optimal performance
- Typical run time is 3 hours
- Creates output files specific to the tool (e.g., resized images)
- Important: The script should be run through
mpi4py.futuresto ensure proper parallel execution.
mpirun -n <num_workers> python -m mpi4py.futures tools_worker.slurmThe source directory structure should follow this pattern:
<path_to_output_folder>/<images_folder>/
├── server_name=<server1>
│ ├── partition_id=<id>
│ │ ├── successes.parquet
│ │ ├── errors.parquet
│ │ └── completed
│ └── partition_id=<id2>
│ ├── ...
└── server_name=<server2>
└── ...
Important components:
- Source folders are partitioned by
server_name=*and thenpartition_id=* - Each partition contains:
successes.parquet: Successfully downloaded imageserrors.parquet: Errors produced during downloadcompleted: Empty marker file indicating the download is complete for this partition
Files will be transferred to a simplified structure:
<dst_image_folder>/
├── server=<server1>
│ ├── data_<uuid1>.parquet
│ ├── data_<uuid2>.parquet
│ └── ...
└── server=<server2>
└── ...
<dst_error_folder>/
├── server=<server1>
│ ├── errors_<uuid1>.parquet
│ ├── errors_<uuid2>.parquet
│ └── ...
└── server=<server2>
└── ...
Key aspects of the output structure:
- Files are organized by server name (with special characters like ':' replaced with '_')
- Each file has a unique UUID to prevent collisions
- Success files are prefixed with
data_and error files witherrors_ - The parquet file format and internal structure from the source is preserved