🟢 dMRI: Practice 2 (~30-minutes + 2 hours job time)

By Dianne K Patterson and Manojkumar Saranathan at the University of Arizona

Practice 2: QSIrecon (~10-minutes)

📋 HPC Batch Job Instructions for QSIrecon

You will modify a script for running QSIrecon on your existing CAM003 data. QSIrecon is included in the qsiprep container.

Caution!

QSIprep has to finish successfully before you can run QSIrecon! Look for the sub-CAM003.html in the derivatives/qsiprep directory and ensure it says No errors to report! at the bottom of the report.

Running QSIrecon will be similar to running QSIprep (For me, qsirecon took 9 hours and 18 minutes!).

✅ Log on to HPC OOD
✅ Navigate to the CAM003_bids_data directory:
Note the path to this directory (Use pwd to display the path), as you'll need it for the runqsirecon.sh script.
✅ You should already have the runqsirecon.sh script to your home bin directory (There are two qsi* scripts, this is the second one you'll work with)

</div> language-'></div>">✅ Open `runqsirecon.sh` in the OOD editor. You are going to modify the account, email and path, like you've done before ✅ Modify the account, e.g., change `--account=dkp` to your own account name, e.g., `--account=akielar`. ✅ Modify `#SBATCH --mail-user=dkp@arizona.edu` to use your email address instead of mine. ✅ Modify the line that begins `export MRIS`. It should point to the directory you just identified with `pwd`: <div class='oc-markdown-activatable oc-markdown-custom-container' data-value='```bash export MRIS=/groups/dkp/Batch/CAM003_bids_data '>

Note

● QSIrecon options vary depending on your analysis goals and what kind of dMRI data you are providing. In this case, it is 3 shell data and I want pyAFQ_tractometry. I have set this in the statement export RECON_SPEC=pyafq_tractometry
● There are other choices for the recon spec, some of which are mentioned but commented out in the script! Note that the time is set to 7 hours, which should accommodate even a lengthy reconstruction.
● Because you have a work directory, you can run other reconstructions and they will re-use what they can from the run directory. This allows you to compare different reconstructions. I suggest making sure you specify a different APP_DERIV_DIR for each (or rename the default qsirecon directory after it has finished being created.
● See QSI Recon Reconstruction Algorithms.

✅ Save your modified batch job.
✅ I suggest switching to elgato:

Run the Batch Job

✅ Make sure you are in the directory where you want the SLURM logs to end up.
✅ Type:

</div> language-'></div>">You should see something like: <div class='oc-markdown-activatable oc-markdown-custom-container' data-value='```bash Submitted batch job 543618 '>

Note

The batch job number is used to track information about the job. You can find it in the Audit tab on the HPC Portal.

✅ Go to the Jobs tab in OOD and see if the job has started yet.
✅ If the job has not failed immediately, then a subdirectory qsirecon will have been created in derivatives and qsirecon_work will have been created in the MRIS directory.. Look to see if they are there.

You do not need to be in interactive mode. The sbatch command will ensure the job is run as a batch job!
You can start the command from any directory, but wherever you start from, the SLURM log will be deposited there. It should have a name similar to this slurm-543618.out. If anything goes wrong (e.g., the qsirecon directory is not generated, or the job finishes in seconds, look in the SLURM log to help troubleshoot. Common problems include not finding the script, not finding the right path or subject, a typo, etc.

Note

● qsirecon should take just over 2 hours to run the pyafq_tractometry reconstruction, and about 1.3 GB of space for the derivatives of one participant. Depending on the reconstruction qsirecon may take considerably more space than this, for example, mrtrix_multishell_msmt_ACT-hsvs reconstruction requires 17 GB of space for the derivatives of one participant!
● If it times out, you can restart it with the exact same command and the job will pick up where it left off (i.e., it'll use the work directory contents so as not to repeat any processing that was already completed).
● The qsirecon_work directory should be about 1.3 GB for the selected pyafq_tractometry reconstruction. Again, a different reconstruction choice can result in a much larger work directory: mrtrix_multishell_msmt_ACT-hsvs reconstruction requires ~25 GB for our participant.

Summary

QSIrecon generates additional files for analysis. You'll examine the output of both qsiprep and qsirecon in the next practice.

What to Expect

Actual run time should be just over 2 hours. Your time allotment is only decremented by the time the job actually takes to run.
SLURM sends you an email about the job. It'll tell you when it starts running, and when the job finishes (ExitCode 0 is a good thing, but just means SLURM was happy with the job, not that the BIDS app succeeded). Importantly, this email also indicates the run time for your job. This can help you adjust the time you request for the job in the future.
A directory derivatives/qsirecon should now exist and contain output (including HTML reports)
The directory qsirecon_work will contain intermediate processing steps, including qsirecon_wf.

Caution! QSIrecon is on the bleeding edge! 07/02/2023

Problems with qsirecon report generation on the HPC can cause reconstruction to fail. To avert this problem, the flag --skip-odf-reports is used. This means the main report that is generated by qsirecon is just text and not very interesting! The developers are aware of the problem, which is some kind of interaction between the Fury viewer and some HPCs.
Note, however, that pyafq_tractometry creates many really nice reports. You'll learn a bit more about these in practice 3.

Exit Questions

TO DO Checklist

✅ When your QSIrecon run finishes, rename the slurm*.out log to slurm_qsirecon.txt and upload it to the D2L qsirecon assignment.
✅ Participate in the Discord channel dmri and share your response to the exit question. Tag your post.

🟢 dMRI: Practice 2 (~30-minutes + 2 hours job time)

Practice 2: QSIrecon (~10-minutes)

📋 HPC Batch Job Instructions for QSIrecon

Run the Batch Job

Summary

What to Expect

Exit Questions

TO DO Checklist

Sharing lessons in OpenClass