Archiving and Data transfer

If you have lots of files and directories, Globus slows to a crawl. For example, a Globus transfer of my ~100 GBs of data: (150 subjects in a directory containing BedpostX results) was less than half done a day later!
Unfortunately, zipping or tarring the datasets takes some time and you’ll have to run such archiving procedures as interactive jobs.
Here is my experience: Tar alone for the 100 GB directory (no zipping) took 7-minutes to archive and resulted in a 92.75 GB file: tar -cvf fsl_dwi_proc.tar fsl_dwi_proc.
Tar with zip (z) took 1 hour and 16 minutes to archive, and resulted in a 90.99 GB file: tar -cvzf fsl_dwi_proc.tar fsl_dwi_proc.
Thus tar+zip saved me only ~2% on file size, but took almost 11 times longer to archive!
Zip by itself took a long time as well. Given that the NIfTI images are all gzipped anyhow, adding z to the tar command just adds time and complexity to the command without much benefit.
Using tar without the z is the way to go for our NIfTI data. After the data is archived, the file took only minutes to transfer, instead of days!

Warning: Archiving operations should be done in interactive mode

If you tar your data from the login note instead of using an interactive desktop or interactive terminal session, you are likely to end up with corrupted data and no warning! Tarring is a compute intensive process and you should expect to use your compute hours for it.

This 3.5-minute video explains the importance of archiving your data before transferring it.

HPC 3 (5 pts): Data Transfer