Globus and Archiving

Globus

The first half of this video is about Globus, which is used by national labs and computing clusters, including the U of A HPC.
Globus transfers data between endpoints (e.g., the HPC and another computer, or even two HPC directories); the endpoints are displayed as panels in a browser window.

The second half of this video is about tar and zip or gzip (these are slightly different).
If you have lots of files and directories (e.g. BedpostX directories), Globus slows to a crawl. For example, a Globus transfer of my ~100 GBs of data: (150 subjects in a directory containing BedpostX results) was less than half done a day later!
Unfortunately, zipping or tarring the datasets take some time and you’ll have to run such archiving procedures as batch jobs.
Here is my experience: Tar alone for the 100 GB directory (no zipping) took 7-minutes to archive and resulted in a 92.75 GB file: tar -cvf fsl_dwi_proc.tar fsl_dwi_proc.
Tar with zip (z) took 1 hour and 16 minutes to archive, and resulted in a 90.99 GB file: tar -cvzf fsl_dwi_proc.tar fsl_dwi_proc.
Thus tar+zip saved me only ~2% on file size, but took almost 11 times longer to archive!
Zip by itself took a long time as well. Given that the NIfTI images are all gzipped anyhow, adding z to the tar command just adds time and complexity to the command without much useful effect.
Using tar without the z is the way to go for our NIfTI data. After the data is archived, the file took only minutes to transfer, instead of days!

This 8-minute video introduces Globus for transferring files and explains the importance of archiving your data before transferring it.