...
Globus is a utility (technically, SaaS, or Software as a Service) which among others allows file transfer between personal computers and HPC clusters. It is particularly useful for the fast and reliable transfer of very large files, as well as a large number of small files.
In order to avail yourself of Globus, you must first create an account (including your user name and password) by going to https://www.globus.org/signup. To transfer data to/from the HPC clusters, submit our request form to join Columbia's Globus subscription group (you will need to log in first - new users should select Columbia University from the drop-down). Email globus@columbia.edu with any questions.
Globus moves data between "endpoints", now called "Collections". An endpoint is a unique name representing a Globus resource like a computer or a cluster, typically in the following format: <globus-username>#<machine name>. The endpoints for the resources you'll be using for transferring data need to be added to your account. After logging into the globus.org website, click on "File Manager" and type "Columbia Dixon" into the "Collections" field. You should then see a "Columbia Dixon" endpoint which should be selected.If you're copying to your own machine
NOTE:
Insomnia's Globus endpoint name: "Insomnia-CUIT"
Before connecting, you will need to create your own endpoint on the machine from which or to which you will be transferring data to and from Free TierInsomnia. In order to do to this, you'll need to download and run 'Globus Connect Personal' as described in:
...
Once you run the downloaded software, you will be able to, via the online interface, enter the two endpoints of your transfer, specify specify the paths of the files/directories on the source and destination systems, and launch the transfer. Once it starts, it will take place in the background and you do not need to supervise it or even be logged in. However, please keep in mind that your access to Free Tier needs to be authenticated (with your regular UNI password), and then possibly re-authenticated at times.
Globus is a fairly sophisticated system that which allows you also to work via Command Line Interface and programming API, and has other useful features. For details, please visit the globusonline.org site.
You can also create public or private Globus file shares to share files or code with colleagues and researchers as illustrated by Globus.
/wiki/spaces/rcs/pages/62139059
...
Copying files from Ginsburg or Terremoto
Users that have an account on the Ginsburg cluster may access Ginsburg's storage system (/burg) by navigating to the /burg directory on the Insomnia login nodes.
Terremoto users can access their storage system (/moto) by navigating to /moto on the Insomnia login nodes.
Note that neither /burg nor /moto are accessible on Insomnia's compute nodes.
Managing Jobs
To view all jobs on the system, use the squeue command.
...
To view fair share information, use sshare.
| Code Block |
|---|
$ sshare
|
Free Tier commands (and their correspondence to Yeti commands)
Slurm commands differ substantially both in syntax and underlying approach from those of Torque/Moab used on Yeti. Here's a very partial list of useful Slurm commands with their roughly corresponding Yeti equivalents (in comments, along with a brief description)
| Code Block |
|---|
sbatch # qsub (submit a job)
|
| Code Block |
|---|
sinfo # qstat -q (list of partitions/queues on the cluster)
|
| Code Block |
|---|
scontrol show job <jobID> # checkjob <jobID> (see status of running job)
|
| Code Block |
|---|
sacct -j <jobID> # tracejob [-n<days>] <jobID> (see status of completed job) |
| Code Block |
|---|
scontrol show node <nodeName> # checknode <nodeName> (information about a node)
|
| Code Block |
|---|
sshare # mdiag -f (information fair share)
|
Each command on both systems is naturally replete with flags and optional arguments which customize its functionality. For reference on Slurm. please refer to:
...