Networked-based shared research data storage is available for internal work for U19 labs as well as cross-lab collaboration needs. The data storage is mounted to a MySQL database hosted on s1n4.u19motor.zi.columbia.edu server. This setting enables to save the scientific metadata of the experiments in the database and the recorded data itself in the mounted storage is provisioned as a single Zuckerman Institute Research Computing Engram storage system Labshare volume. This volume contains one subfolder for each lab, read/write-able only for specified lab members, as well as one shared subfolder, which is read/write-able to all researchers affiliated with Motor U19 projects. The list of subfolders is:
- abbott
- lfa2103 (Larry Abbott)
- bruno
- rb2604 (Randy Bruno)
- churchland
- mc3502 (Mark Churchland)
- costa
- rc3031 (Rui Costa)
- cunningham
- jpc2181 (John Cunningham)
- hillman
- eh2245 (Elizabeth Hillman)
- wx2203 (Weihao Xu)
- jm3648 (Jozsef Meszaros)
- paninski
- lmp2107 (Liam Paninski)
- peterka
- dp2403 (Darcy Peterka)
- lh2881 (Luke Hammond))
- sawtell
- ns2635 (Nate Sawtell)
- shared
- all UNIs for other directories
- zheng
- tz33 (Tian Zheng)
As of March 25, 2019. The stored data has some security measures that will keep the integrity of the database and it is under the general backup policy. Additionally, the Engram storage system is mounted to the database for read-only proposes and can be referred from the database using a file path which has lower level of security in terms of database integrity. DataJoint software is managing the MySQL database using its pipeline managing tools. The storage system integrated with the database and pipeline manager will give the researchers strong and adaptive data management tools.
As of June 1, 2020, the total storage capacity has been set to 44 TB, and each lab subfolder has been configured with a 4 TB "soft" limit. That means if a lab subfolder approaches its 4 TB limit, the lab PI is emailed by the system, and there is a grace period of seven (7) days to correct the situation. The "shared" subfolder has a 4 TB "hard" limit, which means data can not be written to the folder once it is becomes full. The total storage capacity and subfolder configuration can be adjusted as needed, and U19 user account creation can be requested, by emailing support@ziemailing rc@zi.columbia.edu and and specifying "U19-motor database access" in the subject line. Lab PIs approve of U19 user account requests. For assistance please email Yaki Stern, ys3328@columbia.edu.
You can connect Connecting to the shared storage database using this location:\\labshare-smb.engram.rc.zi.columbia.edu\u19motor-labsharePython:
https://tutorials.datajoint.io/beginner/building-first-pipeline/python/first-table.html
Connecting to the database using Matlab:
https://tutorials.datajoint.io/beginner/building-first-pipeline/matlab/first-table.html
For more details regarding DataJoint please see : https://docs.datajoint.io/
General information about connecting to the Engram storage system is available here.
...