Transfers from Engram to/from RCS clusters

The ITS team provides two methods to facilitate data transfers between the RCS Clusters and Engram:

Engram Transfer Server: engram-xfer-01.rc.zi.columbia.edu
Globus: For local shares and streamlined transfer.

Method 1: Engram Transfer Server

Server Address: engram-xfer-01.rc.zi.columbia.edu

Purpose: A dedicated Linux server where users can mount Engram shares and transfer files to/from external servers (like the RCS clusters) without utilizing local workstation resources

Users please skip to the second step "Transfer Process"

System Admins only, not for users!

Account Setup: Contact ITS to set up a user account. Provide the Engram share you need access to and any UNIs (Columbia University Network IDs) requiring access.
Permissions & Access:
- Authentication: Server uses CU LDAP for UNI authentication. Verify if the new user exists in LDAP using id UNI. If not, request CUIT to create a UNI account.
- Add User to Access Group:
  sudo usermod -aG engram-xfer-01 UNI
- Create Home Directory:
  sudo mkdir /home/UNI sudo chown UNI:user /home/UNI sudo chmod 700 /home/UNI
- CIFS Mount Script Permissions: Grant users permission to run Engram mount scripts:
  - /usr/local/bin/zi_smbmount
  - /usr/local/bin/zi_smbunmount
- Lab Group Permissions:
  - Confirm or create the lab group in the system:
    grep -R LabName /etc/group grep -R LabName /etc/sudoers
  - If the lab group doesn’t exist, create it and set permissions:
    sudo groupadd LabName echo "%LabName ALL=/usr/local/bin/zi_smbmount LabName-EngramShareTier, /usr/local/bin/zi_smbunmount" >> /etc/sudoers
  - Add New User to Lab Group:
    sudo usermod -a -G LocalLabName NewUser

Transfer Process:

Make sure you are on one of the three networks below when you are connecting the server and you should use your UNI and UNI passwordL
1. Columbia secure wifi
2. Columbia VPN
3. Ethernet cable physically plugged in wall jack in JLG
Login:
- Connect to engram-xfer-01.rc.zi.columbia.edu:
  ssh UNI@engram-xfer-01.rc.zi.columbia.edu
- Use Columbia Secure WiFi, a campus network jack, or VPN.
Mounting Engram Share:
- Create Mount Point:
  mkdir /home/UNI/mnt
- Mount Share:
  /usr/local/bin/zi_smbmount LabName-EngramShareTier
- Unmounting and Remounting: Run /usr/local/bin/zi_smbunmount to unmount and remount as needed.
Data Transfer Using rsync:
- Start a tmux session to allow persistent transfer even after disconnecting.
  tmux
- Rsync Commands:
  - From Engram to RCS:
    rsync -avPhW /home/UNI/mnt/source_path UNI@ginsburg.rcs.columbia.edu:/destination_path
  - From RCS to Engram:
    rsync -avPhW UNI@ginsburg.rcs.columbia.edu:/source_path /home/UNI/mnt/destination_path
- Monitoring: Reattach the tmux session:
  bash
  tmux attach -t SessionName
- Resume Failed Transfers:
  bash
  rsync -avhW SourceDirectory Destination

Additional Resources:

Method 2: Globus Transfers

Globus Setup:
ITS can set up a Globus local share for you, allowing data transfers through the Globus tool without burdening your workstation.

To Request: Contact ITS for assistance in setting up a Globus share.

Zuckerman Institute

Transfers from Engram to/from RCS clusters

Analytics

Method 1: Engram Transfer Server

Method 2: Globus Transfers

Related content