Overview

To better serve our research community who require high data usage in HPC environment to run ML/AI kind of jobs, DSBIT is re-vamping the storage to ultra fast NVMe based Weka storage.

WekaFS is a fully distributed parallel filesystem leveraging NVMe Flash for file services. Integrated tiering seamlessly expands the namespace to and from HDD object storage, simplifying data management. WekaFS stands out with its unique architecture, overcoming legacy systems’ scaling and file-sharing limitations. Supporting POSIX, NFS, SMB, S3, and GPUDirect Storage.

What’s NEW in storage?

Users (Previously home)

• Exclusive storage space of 20GB assigned to each C2B2 Cluster user.

• Used for small input files, source code files and executables, software builds etc.

• POSIX/NFS shared

• Backed up nightly.

Groups (Previously data)

• Shared space of 1TB storage allocated by default to each PI_group.

• General space for group software, common data, etc.

• POSIX/NFS shared

• Backed up weekly

Archive

• Shared space for each group.

• PI can purchase spinning disk storage to store their data for long term.

• SMB mounted only, can be accessed via Globus

• Backed up monthly to another S3 block storage device (in the same building but different floor)

NOTE:

Scratch storage tier is not offered in Weka, although PI can create a directory called Scratch to temporarily copy data required to run their jobs under groups/PI_gp storage tier.

Groups and Archive tier storage are available in capacity increments of 1TB.

Pricing details for each of the tiers will be shared once the New Cluster is launched.

For any questions/comments, please send an email to dsbit_help@cumc.columbia.edu