Globus at C2B2

All C2B2 storage systems are configured as Globus endpoints to facilitate the transfer of data. As an alternative to SFTP, Globus is a service that makes it easy to move, sync, and share large amounts of data. Globus will manage file transfers, monitor performance, retry failures, recover from faults automatically when possible, and report the status of your data transfer.

Globus can be used in a number of different ways:

• Transfer large amounts of data between C2B2 storage systems (e.g. Isilon Data/Archival to HPC/Weka or vice versa)

• Transfer large amounts of data from C2B2 storage systems to a personal computer with Globus Connect Personal

• Transfer large amounts of data between C2B2 storage systems and storage systems at other institutions.

• Publish and curate data sets with Globus Data Publication

• Set up automated transfers using the Globus CLI utility or your own application/script using the Globus APIs

This step-by-step guide will show you how to log into Globus and use it to transfer files reliably and securely.

C2B2 Endpoint Reference

The Globus endpoint for transferring data to or from your Isilon /ifs/home, /ifs/data /ifs/scratch or /ifs/archive areas is C2B2-STORAGE. You do not need to be logged on to HPC cluster or Isilon to start or monitor a transfer

Login with an existing identity

Columbia University users need to login with COLUMBIA credentials to access Globus. Visit http://www.globus.org and click "Login" at the top of the page. On the Globus login page, choose an organization Columbia University, and then click "Continue." You’ll be redirected to familiar looking page for Columbia login. Use your Columbia UNI and password to login.

Open

The File Manager

After you’ve logged in to Globus, you’ll begin at the File Manager. The first time you use the File Manager, all fields will be blank.

Open

Access a collection

Click in the Collection field at the top of the File Manager page and type the name of the endpoint. Globus will list collections with matching names. The collection hosted by C2b2 start with "C2B2-STORAGE"

Open

Click on the Globus Endpoint C2B2-STORAGE. Globus will connect to the collection and display the default directory, /. Click the "Path" field and change it to the directory containing the data to be transferred.

Open

Request a file transfer

Click "Transfer or Sync to..." in the command panel on the right side of the page, or change the "Panels" view at the top right of the File Manager page. This will show both panels for both endpoints. Select the "Transfer or Sync to" field to select the endpoint that will receive the data.

Open

Find the second endpoint collection and connect to it as you did with the first endpoint above. Navigate to the receiving location on the second endpoint.
Between the two Start buttons at the bottom of the page, the Transfer & Sync Options tab provides access to several options. By default, Globus verifies file integrity after transfer using checksums. Click the information icons for explanations of the other transfer settings. Globus gives you powerful control over the behavior of the transfer with a simple mouse click. Change the transfer settings if you’d like. You may also enter a label for the transfer, but this isn’t required.
Click the Start> button to transfer the selected files to the collection in the right panel. Globus will display a green notification panel, confirming that the transfer request was submitted.

Open

Confirm transfer completion

Click Activity in the command menu on the left of the page to go to the Activity page. On the Activity page, click the arrow icon on the right to view details about the transfer. You will also receive an email with the transfer details.

Open

Email Confirmation for Transfer

Open

Does Globus have sync option like rsync?

Yes, it does the sync option to update only new and modified files.

Here's a breakdown of the sync options and how to use them:

1. Sync - Only transfer new or changed files:

• This option allows you to transfer only the files that are new or have been modified since the last transfer. 

• It helps to save time and bandwidth by avoiding redundant transfers. 

• The checksum is used to determine if a file has changed. 

• You can select this option in the "Transfer & Sync Options" menu. 

2. Verify file integrity after transfer:

• This option ensures that the files transferred are identical to the source files. 

• Globus uses checksums to verify the integrity of the files. 

• It is recommended to enable this option to ensure data integrity. 

3. Delete files on destination that do not exist on source: 

• This option deletes any files in the destination directory that do not exist in the source directory.

• It is helpful for maintaining a clean destination directory, but it should be used with caution as it can lead to data loss if not used carefully.

4. Other Sync Options: 

• Preserve source file modification times:

  .Opens in new tab

  This option ensures that the modification times of the files are preserved during the transfer.

• Encrypt transfer:

  .Opens in new tab

  This option allows you to encrypt the transfer of data, providing an extra layer of security.

How to access and use these options:

1. Access the Transfer & Sync Options menu: In the Globus web interface, navigate to the "Transfer & Sync Options" menu located between the two transfer columns.

2. Select the desired options: Check the boxes next to the options you want to enable.

3. Start the transfer: Click the "Start" button to initiate the transfer with the selected options. 

Below is the screenshot shows the setting option:

Open