Versions Compared

Version Old Version 11 New Version Current
Changes made by

George Garrett

Yichao Zhang

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Engram Transfer Server: engram-xfer-01.rc.zi.columbia.edu
  2. Globus: For local shares and streamlined transfer.

...

Method 1: Engram Transfer Server

Server Address: engram-xfer-01.rc.zi.columbia.edu

Purpose: A dedicated Linux server where users can mount Engram shares and transfer files to/from external servers (like the RCS clusters) without utilizing local workstation resources.To Get Started:

Users please skip to the second step "Transfer Process"

...

Warning
titleSystem Admins only, not for users!
  • Account Setup: Contact ITS to set up a user account. Provide the Engram share you need access to and any UNIs (Columbia University Network IDs) requiring access.
  • Permissions & Access:
    • Authentication: Server uses CU LDAP for UNI authentication. Verify if the new user exists in LDAP using id UNI. If not, request CUIT to create a UNI account.
    • Add User to Access Group:

...

    • sudo usermod

...

    • -aG engram-xfer-01 UNI
    • Create Home Directory:

...

    • sudo mkdir /home/UNI sudo chown UNI:user /home/UNI sudo chmod 700 /home/UNI
    • CIFS Mount Script Permissions: Grant users permission to run Engram mount scripts:
      • /usr/local/bin/zi_smbmount
      • /usr/local/bin/zi_smbunmount
    • Lab Group Permissions:
      • Confirm or create the lab group in the system:

...

      • grep -R LabName /etc/group grep -R LabName /etc/sudoers
      • If the lab group doesn’t exist, create it and set permissions:

...

      • sudo groupadd LabName echo "%LabName ALL=/usr/local/bin/zi_smbmount LabName-EngramShareTier, /usr/local/bin/zi_smbunmount" >> /etc/sudoers
      • Add New User to Lab Group:

...

      • sudo usermod -a -G LocalLabName NewUser


Transfer Process:

  1. Make sure you are on one of the three networks below when you are connecting the server and you should use your UNI and UNI passwordL
    1. Columbia secure wifi
    2. Columbia VPN
    3.  Ethernet cable physically plugged in wall jack in JLG

  2. Login:

  3. Mounting Engram Share:

    • Create Mount Point:
      bash
      mkdir /home/UNI/mnt
    • Mount Share:
      bash
      /usr/local/bin/zi_smbmount LabName-EngramShareTier
    • Unmounting and Remounting: Run /usr/local/bin/zi_smbunmount to unmount and remount as needed.

  4. Data Transfer Using rsync:

    • Start a tmux session to allow persistent transfer even after disconnecting.
      bash
      tmux
    • Rsync Commands:
    • Monitoring: Reattach the tmux session:
      bash
      tmux attach -t SessionName
    • Resume Failed Transfers:
      bash
      rsync -avhW SourceDirectory Destination

Additional Resources:

...

Method 2: Globus Transfers

Globus Setup:
ITS can set up a Globus local share for you, allowing data transfers through the Globus tool without burdening your workstation.

...