Overview

To better serve our research community who require high data usage in HPC environment to run ML/AI kind of jobs, DSBIT is re-vamping the storage to ultra fast NVMe based Weka storage.

WekaFS is a fully distributed parallel filesystem leveraging NVMe Flash for file services. Integrated tiering seamlessly expands the namespace to and from HDD object storage, simplifying data management. WekaFS stands out with its unique architecture, overcoming legacy systems’ scaling and file-sharing limitations. Supporting POSIX, NFS, SMB, S3, and GPUDirect StorageIn response to ever increasing requirements for lower-latency and higher throughput file services to support modern AI/ML workloads, we have added an all-NVMe based WekaFS tier to our storage profile. This highest performance parallel filesystem is directly attached to our new HPC cluster serving I/O over HDR200/100 InfiniBand. The existing Isilon NAS has been moved exclusively to Campus-wide SMB service for the labs and desktops.

All storage systems, lab instruments, and collaborating institutions are connected via GLOBUS, a de-facto standard for data sharing platform among all premier Universities, research Labs and National data repositories. GLOBUS provides very secure & efficient data movements between any GLOBUS-connected end-point, within a room to across the globe, from the convenience of a web browser. This allows users to plan & move their data between Weka and Isilon systems, or anywhere else, in preparation for running their HPC jobs.

What’s NEW in storage?

...