Page Comparison

ITS The ITS team provides two methods to transfer facilitate data transfers between the RCS Clusters and Engram:

1. Engram Transfer Server: engram-xfer-01.rc.zi.columbia.edu
2. We can set up user an account on this linux server that Globus: For local shares and streamlined transfer.

...

Method 1: Engram Transfer Server

Server Address: engram-xfer-01.rc.zi.columbia.edu

Purpose: A dedicated Linux server where users can mount Engram shares and

...

transfer

...

files to/from

...

external servers

...

(like the RCS

...

clusters) without utilizing local workstation resources.

To Get Started:

• Account Setup: Contact ITS to set up a user account. Provide the Engram share you need access to and any UNIs (Columbia University Network IDs) requiring access.
• Permissions & Access:
  • Authentication: Server uses CU LDAP for UNI authentication. Verify if the new user exists in LDAP using id UNI. If not, request CUIT to create a UNI account.
  • Add User to Access Group:
    bash

    sudo usermod -m -aG engram-xfer-01 UNI

...

• • Create Home Directory:
    bash

    sudo mkdir /home/UNI

...

• • sudo chown UNI:user /home/UNI

...

• • sudo chmod 700 /home/UNI

...

...

• • CIFS Mount Script Permissions: Grant users permission to run

...

• • Engram mount

...

• • scripts

...

• • :
    • /usr/local/bin/zi_smbmount

...

• • • /usr/local/bin/zi_smbunmount

...

• • Lab Group Permissions:
    • Confirm or create the lab

...

• • • group

...

• • • in the system:
      bash

      grep -R LabName /etc/group

...

• • • grep -R LabName /etc/sudoers

...

• • • If the lab group

...

• • • doesn’t exist, create it and set permissions:
      bash

      sudo groupadd LabName echo "%LabName ALL=/usr/local/bin/zi_smbmount LabName-EngramShareTier, /usr/local/bin/zi_smbunmount" >> /etc/sudoers
    • Add

...

• • • New User to Lab Group:
      bash

      sudo usermod -a -G LocalLabName NewUser

...

Transfer Process:

1. Login:

   • Connect to engram-xfer-01.rc.zi.columbia.edu:
     bash

     ssh UNI@engram-xfer-01.rc.zi.columbia.edu
   . You have to make sure you are either on
   • Use Columbia Secure
   Wifi
   • WiFi, a
   on campus computer that plugged in a wall jack or on Columbia's VPN network.Create a Engram share mount point on this sever by running "
   • campus network jack, or VPN.

2. Mounting Engram Share:

   • Create Mount Point:
     bash

     mkdir /home/UNI/mnt
   "
   • Mount
   your lab's Engram share on this server: "
   • Share:
     bash

     /usr/local/bin/zi_smbmount LabName-EngramShareTier
   " (eg,  
   • Unmounting and Remounting: Run /usr/local/bin/zi_
   smbmount shohamy-locker ). Please note: you only need to do ii and iii steps once unless you unmount the engram share by running "/usr/local/bin/zi_smbmount". Then you need to run "/usr/local/bin/zi_smbmount LabName-EngramShareTier" again when you need Engram mounted on this server.Transfer data between HPC and Engram (engram-xfer-01.rc):
3. Open a new tmux session by running "tmux" or "tmux new -s SessionName". You  can also rename the tmux by press keyboard buttons: ctrl + b, $ or reattach a existing tmux session by running : "tmux attach -t SessionName".
Run the command to transfer data between HPC and Engram:  rsync -avPhW  SourceDirectory Destination. Example for transferring data from Engram to Ginsburg:
• smbunmount to unmount and remount as needed.

4. Data Transfer Using rsync:

   • Start a tmux session to allow persistent transfer even after disconnecting.
     bash

     tmux
   • Rsync Commands:
     • From Engram to RCS:
       bash

       rsync -avPhW /home/
   xxx
   • • UNI/mnt/
   path/to/source/data 
   • • source_path UNI@ginsburg.rcs.columbia.edu:
   /directory
   • • /
   to/destination/directory    Example for transferring data from Ginsburg
   • • destination_path
     • From RCS to Engram:
    rsync
   • • bash

       rsync -avPhW UNI@ginsburg.rcs.columbia.edu:/
   directory/to/destination/directory /
   • • source_path /home/
   xxx
   • • UNI/mnt/
   path/to/source/data. You can also ssh on ginsburg and run rsync data from ginsburg. Example for transferring data from Engram to Ginsburg on Ginsburg: rsync -avPhW  /path/to/source/data  UNI@engram-xfer-01.rc.zi.columbia.edu: /path/to/destination. Vice versa.
5. Exit tmux session buy closing the sell window. DO NOT press ctrl c or type "exit". Then you can go and do other stuff, the rsync job will keep running in the air.
6. In order to verify if the rsync job is finished, ssh back to the server you executed the rsync job, reattach the tmux session by running "tmux a -t SessionName".
7. If the job was failed in the middle of the transferring, you can run a command to resume the rsync job when it dropped: sync -avhW  SourceDirectory Destination.  Here are the  links of  detail  usage  of  tumx  and  rsync:  tmux cheat sheet; rsync cheat sheet.
8. Globus

...

1. • • destination_path
   • Monitoring: Reattach the tmux session:
     bash

     tmux attach -t SessionName
   • Resume Failed Transfers:
     bash

     rsync -avhW SourceDirectory Destination

Additional Resources:

• tmux Cheat Sheet
• rsync Cheat Sheet

...

Method 2: Globus Transfers

Globus Setup:
ITS can set up a Globus local share for you, allowing data transfers through the Globus tool without burdening your workstation.

To Request: Contact ITS for assistance in setting up a Globus share.