Transfers from Engram to/from RCS clusters

The ITS team provides two methods to facilitate data transfers between the RCS Clusters and Engram:

1. Engram Transfer Server: engram-xfer-01.rc.zi.columbia.edu
2. Globus: For local shares and streamlined transfer.

Method 1: Engram Transfer Server

Server Address: engram-xfer-01.rc.zi.columbia.edu

Purpose: A dedicated Linux server where users can mount Engram shares and transfer files to/from external servers (like the RCS clusters) without utilizing local workstation resources.

To Get Started:

• Account Setup: Contact ITS to set up a user account. Provide the Engram share you need access to and any UNIs (Columbia University Network IDs) requiring access.
• Permissions & Access:
  • Authentication: Server uses CU LDAP for UNI authentication. Verify if the new user exists in LDAP using id UNI. If not, request CUIT to create a UNI account.
  • Add User to Access Group:
    sudo usermod -m -aG engram-xfer-01 UNI
  • Create Home Directory:
    sudo mkdir /home/UNI sudo chown UNI:user /home/UNI sudo chmod 700 /home/UNI
  • CIFS Mount Script Permissions: Grant users permission to run Engram mount scripts:
    • /usr/local/bin/zi_smbmount
    • /usr/local/bin/zi_smbunmount
  • Lab Group Permissions:
    • Confirm or create the lab group in the system:
      grep -R LabName /etc/group grep -R LabName /etc/sudoers
    • If the lab group doesn’t exist, create it and set permissions:
      sudo groupadd LabName echo "%LabName ALL=/usr/local/bin/zi_smbmount LabName-EngramShareTier, /usr/local/bin/zi_smbunmount" >> /etc/sudoers
    • Add New User to Lab Group:
      sudo usermod -a -G LocalLabName NewUser

Transfer Process:

1. Login:

   • Connect to engram-xfer-01.rc.zi.columbia.edu:
     ssh UNI@engram-xfer-01.rc.zi.columbia.edu
   • Use Columbia Secure WiFi, a campus network jack, or VPN.

2. Mounting Engram Share:

   • Create Mount Point:
     mkdir /home/UNI/mnt
   • Mount Share:
     /usr/local/bin/zi_smbmount LabName-EngramShareTier
   • Unmounting and Remounting: Run /usr/local/bin/zi_smbunmount to unmount and remount as needed.

3. Data Transfer Using rsync:

   • Start a tmux session to allow persistent transfer even after disconnecting.
     tmux
   • Rsync Commands:
     • From Engram to RCS:
       rsync -avPhW /home/UNI/mnt/source_path UNI@ginsburg.rcs.columbia.edu:/destination_path
     • From RCS to Engram:
       rsync -avPhW UNI@ginsburg.rcs.columbia.edu:/source_path /home/UNI/mnt/destination_path
   • Monitoring: Reattach the tmux session:
     bash

     tmux attach -t SessionName
   • Resume Failed Transfers:
     bash

     rsync -avhW SourceDirectory Destination

Additional Resources:

• tmux Cheat Sheet
• rsync Cheat Sheet

Method 2: Globus Transfers

Globus Setup:
ITS can set up a Globus local share for you, allowing data transfers through the Globus tool without burdening your workstation.

To Request: Contact ITS for assistance in setting up a Globus share.