Overview
In response to ever increasing requirements for lower-latency and higher throughput file services to support modern AI/ML workloads, we have added an all-NVMe based WekaFS tier to our storage profile. This highest performance parallel filesystem is directly attached to our new HPC cluster serving I/O over HDR200/100 InfiniBand. The existing Isilon NAS has been moved exclusively to Campus-wide SMB service for the labs and desktops.
...
Storage Tiers | C2B2 Cluster | New Cluster |
|---|---|---|
home | home/PI_gp/userUNI | users/userUNI |
data | data/PI_gp/userUNI | groups/PI_gp/userUNI |
archive | archive/PI_gp/userUNI | archive/PI_gp/userUNI |
Users (Previously home)
...
No matter how fast filesystem you have, there is VERY HIGH overhead in opening a file for read or write ops. This will certainly add up if you have large number of small files, and you will be surprised how bad it could get. We cannot emphasize enough that one should avoid creating large number of small files at all cost!!! In addition, use only plain-English characters, without spaces, without special characters, to name the files & folders.
Groups and Archive tier storage are available in capacity increments of 1TB.
...