Networked-based shared research data storage is available for internal work for U19 labs as well as cross-lab collaboration needs. The data storage is mounted to a MySQL database hosted on s1n4.u19motor.zi.columbia.edu server. This setting enables to save the scientific metadata of the experiments in the database and the recorded data itself in the mounted storage is provisioned as a single Zuckerman Institute Research Computing Engram storage system Labshare volume. This volume contains one subfolder for each lab, read/write-able only for specified lab members, as well as one shared subfolder, which is read/write-able to all researchers affiliated with Motor U19 projects. The list of subfolders is:
- abbott
- bruno
- churchland
- costa
- hillman
- paninski
- sawtell
- shared
As of March 25, 2019. The stored data has some security measures that will keep the integrity of the database and it is under the general backup policy. Additionally, the Engram storage system is mounted to the database for read-only proposes and can be referred from the database using a file path which has lower level of security in terms of database integrity. DataJoint software is managing the MySQL database using its pipeline managing tools. The storage system integrated with the database and pipeline manager will give the researchers strong and adaptive data management tools.
As of June 1, 2020, the total storage capacity has been set to 32 TB, and each lab subfolder has been configured with a 4 TB "soft" limit. That means if a lab subfolder approaches its 4 TB limit, the lab PI is emailed by the system, and there is a grace period of seven (7) days to correct the situation. The "shared" subfolder has a 4 TB "hard" limit, which means data can not be written to the folder once it is becomes full.The total storage capacity and subfolder configuration 44 TB. The total storage capacity can be adjusted as needed, and U19 user account creation can be requested, by emailing support@ziemailing rc@zi.columbia.edu and and specifying "U19-motor database access" in the subject line. Lab PIs approve of U19 user account requests.
You can connect to the shared storage using this location:
\\labshare-smb.engram.rc.zi.columbia.edu\u19motor-labshare. For assistance please email Yaki Stern, ys3328@columbia.edu.
Connecting to the database using Python:
https://tutorials.datajoint.io/beginner/building-first-pipeline/python/first-table.html
Connecting to the database using Matlab:
https://tutorials.datajoint.io/beginner/building-first-pipeline/matlab/first-table.html
For more details regarding DataJoint please see : https://docs.datajoint.io/
General information about connecting to the Engram storage system is available here.