Working with External Datasets

Compute nodes on the HPC do have access to the external internet, so datasets can be downloaded directly from within a job if needed. However, for large or frequently reused datasets, it is generally recommended to download the data ahead of time from a login node rather than as part of a scheduled job. This avoids tying up compute allocation on network transfers and reduces the risk of job failures due to slow or interrupted downloads.

There are two filesystems within the HPC that could hold the data (/WAVE/projects/ and /WAVE/datasets/). The primary difference between the two filesystems has to do with backup and recovery. The project's filesystem is backed up on a daily basis, whereas the datasets filesystem is not. The datasets filesystem is appropriate for large external datasets, which can be recovered by downloading a fresh copy from the external source.