| Table of Contents | ||
|---|---|---|
|
...
You must use SCP (secure copy protocol) to transfer data and other program files between your local machine and TerremotoGinsburg. Unix/Linux and Cygwin users can use the scp command, Windows users can use WinSCP, and Mac users can use Fugu. For large file transfers we recommend the use of Globus, documented below.
Please use the interface quakeserver motion.rcs.columbia.edu when transferring files.
...
| Code Block |
|---|
$ scp MyDataFile <UNI>@quake<UNI>@motion.rcs.columbia.edu:<DESTINATION_PATH> |
...
If you specify no path after the colon, the file will end up in your home directory (no check for the existence of its older version is performed in this case).
Globus Connect (Under Construction)
Globus is a utility (technically, SaaS, or Software as a Service) which among others allows file transfer between personal computers and HPC clusters. It is particularly useful for fast and reliable transfer of very large files, as well as a large number of small files.
...
In order to avail yourself of Globus, you must first create an account (including your user name and password) by going to https://www.globus.org/signup. Globus moves data between "endpoints". An endpoint is a unique name representing a Globus resource like a computer or a cluster, typically in the following format: <globus-username>#<machine name>. The endpoints for the resources you'll be using for transferring data need to be added to your account.To use Terremoto
| Note |
|---|
Ginsburg's Globus endpoint |
...
has not been activated yet. Once it is available we will share information here on how to connect to it. |
Before connecting, you will need to create your own endpoint on the machine from which or to which you will be transferring data to and from TerremotoGinsburg. In order to to this, you'll need to download and run 'Globus Connect Personal' as described in:
...
Globus is a fairly sophisticated system which allows you also to work via Command Line Interface and programming API, and has other useful features. For details, please visit the globusonline.org site.
Copying files from Habanero or Terremoto
Users that have an account on the Habanero cluster may access Habanero's storage system (/rigel) by navigating to the /rigel directory on Ginsburg login nodes.
Terremoto login nodes and transfer nodes (shake, bake, and quake). Habanero's storage system is not accessible on the users can access their storage system (/moto) by navigating to /moto on the Ginsburg login and transfer node.
Note that neither /rigel nor /moto are accessible on Ginsburg's compute nodes.
Managing Jobs
...
Note that existing user sessions will not be affected by changes in group membership. If a user is added or removed from a group they will have to log out and log back in to HabaneroGinsburg. Note also that in some cases group changes can take up to an hour to propagate to HabaneroGinsburg.
There are many ways to view your group membership. One way is to use the groups command.
| Code Block |
|---|
$ groups user yetiapamburglab apam |
Once a group has been created and users have been added, group members can use the chgrp and chmod commands to set group ownership and permissions on files and directories to values appropriate for sharing. For more information see the manual pages for the two commands.
...
| Code Block |
|---|
$ man chgrp $ man chmod |
Open OnDemand Web Portal Access
Terremoto users may also access the cluster and submit jobs via the Open OnDemand web portal.
Open OnDemand is a web portal service developed by the Ohio Supercomputer Center that provides users with access to the Terremoto cluster and its underly file system. It allows to view, edit, upload and download files, create, edit, submit and monitor jobs, and connect via SSH, all via a web browser and with a minimal knowledge of Linux and scheduler commands.
Connecting to Open OnDemand
To connect, point your web browser to https://quake.rcs.columbia.edu and authenticate with your Columbia UNI and UNI password. After successfully logging in you will come to a dashboard page that looks like this:
From this menu you can access your home directory files as well as submit and monitor jobs.
PLEASE NOTE: a known existing issue with Open OnDemand is that Open OnDemand will not log you out until you have fully shutdown your browser. Because of this we discourage the user of Open OnDemand from public computers and encourage you to shut down your browser upon completing your work. Another option may be to use Open OnDemand from a "private" browsing window or "Incognito" window. We plan on implementing a fix for this once it has been released by the Open OnDemand developers.
File Management
The Files menu allows one to view and operate on files in user's home directory. This page will show your available files and folders and provide tools to view and download current files, create new files, and upload files to your home directory.
Job Composer
Jobs can be created, edited and submitted with the job management tools under the Jobs menu with the Job Composer. The Job Composer serves as a GUI alternative to Slurm scheduler commands and allows to write Slurm batch scripts and create script templates.
Users can create a new job either from the default job template - a mostly empty Slurm batch script file - or providing the path to an existing slurm batch script.
Users may have to open the Job Options menu and enter their Account membership, like shown below.
The Account can also be included in the Slurm job script, which can be edited either from the command line or directly in the GUI interface:
Job Monitoring
Users may monitor their jobs that are currently running via the Jobs menu by selecting Active Jobs
Shell Access
Users can open a command line interface by selecting the Clusters menu and then the "terremoto Shell Access" option. This will open a command line interface directly in your browser, rather than needing to connect via the terminal or PuTTY program.
...