Introduction

Goals To learn about batch processing on the HPC so you can run big jobs unattended

Learning Objectives This lesson contributes to the objective to "operate the High-Performance Computing Resources to transfer data, submit batch jobs, and work with Singularity/Apptainer containers" by having you implement "HPC SLURM batch jobs" and "HPC SLURM batch array jobs".

Practice Overview

This practice consists of two tasks:

1) Practice 1 Run a batch job that calls a BIDS neuroimaging container. It is small and fast, and one of the earliest examples of a BIDS app.

2) Practice 2 Run a batch array job. An array job runs the same process on many datasets, and it is extremely efficient.

⛅️ HPC Batch Job Tutorial

In the previous lesson you uploaded a dataset. In this practice lesson you are going to run a batch job that calls bet (FSL's brain extraction tool). This only takes a few minutes to run: See Neuroimaging core read-the-docs: BET container.

Watch this 6.75-minute video demonstration

📋 HPC Batch Job Instructions

Dataset

You will need the CAM003_bids_data.zip (226 MB) dataset from OSF.  You should have this in your home directory because you uploaded it previously with Globus.

✅ Move CAM003_bids_data.zip to your lab's group or xdisk directory. 

✅ Unzip the dataset (this dataset is small, you don't need interactive mode):

unzip CAM003_bids_data.zip

✅ Rename the dataset to mri_practice. Navigate to the directory:

mv CAM003_bids_data mri_practice
cd mri_practice

The mri_practice directory contains the fully BIDS compliant subdirectory data with a single subject, sub-CAM003.  

Script

runbet.sh is available from /groups/dkp/neuroimaging/scripts.

✅ If you do not already have runbet.sh in your ~/bin directory, copy it now:

cp /groups/dkp/neuroimaging/scripts/runbet.sh ~/bin

✅ Confirm that runbet.sh has executable permissions (x)

ls -l ~/bin 
-rwxr-xr-x. 1 dkp dkp     1514 Dec 15  2022 runbet.sh

✅ Ensure your copy of runbet.sh is in the path 

which runbet.sh
~/bin/runbet.sh

✅ Open the batch job file runbet.sh in the OOD editor (runbet.sh should be in your home bin directory AND it should be executable):

✅ In the open runbet.sh file, modify the account, e.g., change --account=dkp to your own account name, e.g., --account=akielar.

✅ Modify #SBATCH --mail-user=dkp@arizona.edu to use your email address instead of mine.

✅ Modify the line that begins export MRIS. It should point to the mri_practice directory you just identified with pwd. For example, my test run now reads:

export MRIS=/groups/dkp/Batch/mri_practice

Run the Batch Job

✅ Type:

sbatch --export sub=CAM003 runbet.sh

✅ Go to the Jobs tab in OOD (you only have a few minutes, unless the job gets queued).

• You do not need to be in interactive mode. The sbatch command will ensure the job is run as a batch job!

• You can start the command from any directory, but wherever you start from, the SLURM log (e.g., slurm-542994.out) will be deposited there. If anything goes wrong (e.g., the derivatives directory is not generated, or the brain file is not created), look in the SLURM log to help troubleshoot. Common problems include not finding the script, not finding the right path or subject, a typo, etc.

• The bet container was one of the earliest BIDS apps. It uses an ancient copy of the BIDS validator internally and does not recognize many files that are now recognized, so it will complain about some of the files in this dataset. Nevertheless, it should run and create sub-CAM003_ses-01brain_brain.nii.gz under the derivatives directory.

• Actual run time (not queued time) should be less than 1 minute. Your time allotment is only decremented by the time it actually took to run the job.

• SLURM sends you email about the job. When the job finishes, ExitCode 0 is a good thing, but just means SLURM was happy with the job (i.e., SLURM found zero problems), not that the BIDS app succeeded. Importantly, this email also indicates the run time for your job. This can help you adjust the time you request for the job in the future.

• Congratulations! If this worked, you have now run a batch job on BIDS neuroimaging data calling a container to do the work! If it didn't work, be patient with yourself and try to track down the problem. Remember I'm around if you get frustrated.

✅ Move the slurm output to a .txt file, e.g.:

mv slurm-542994.out slurm-bet.txt

You will copy and paste its contents into the question at the end of this lesson.

⛅️ HPC Array Job Tutorial: sbatchr and array jobs

sbatchr is a wrapper script for running typical BIDS processing, it is not an official HPC script. sbatchr will create a SLURM array job for each subject in a list of subjects you pass it (one subject per row in a text file).

✅ Watch the 4-minute demonstration video and follow the instructions below.

📋 sbatchr and array job setup

✅ Open a Bash terminal.

✅ Copy sbatchr and array.sh from /groups/dkp/neuroimaging/scripts to your ~/bin directory:

cp /groups/dkp/neuroimaging/scripts/{sbatchr,array.sh} ~/bin

✅ In the OOD "Files" tab, switch to your home bin directory. Open your copy of array.sh with the OOD editor.

✅ Modify the account name from dkp to your own lab account name. That’s it! The script is ready to go.

✅ Create a practice directory (it is fine if this is in your home directory)

✅ Open your empty my_subjects.txt file with the OOD editor and add some values, e.g.,:

mkdir array_practice; cd array_practice

✅ Switch to the OOD: Active Jobs view, so you can see all of the jobs have been completed (this is really fast, so you have to catch it before it disappears). Using this same approach, you could run something more complex, like MRIQC or fMRIPrep instead of touch.

📋 sbatchr and array jobs instructions

✅ Run the batch job:

sbatchr array.sh mysubjects.txt

Finished SLURM Array Job

✅ When it is done, you’ll see a SLURM array job file for each of your subjects:

ls -1 array*
array_slurm_my_subjects-1.out
array_slurm_my_subjects-2.out
array_slurm_my_subjects-3.out
array_slurm_my_subjects-4.out
array_slurm_my_subjects-5.out

✅  Use cat to view the contents of one of these jobs, e.g.:

cat array_my_subjects-1.out 

/home/u7/dkp/array_practice
r2u07n1.puma.hpc.arizona.edu
Thu Dec 15 12:55:09 MST 2022
JOBNAME=array_my_subjects, JOB ID: 5909957, Array Index: 1, Subject: sub-014
Detailed performance metrics for this job will be available at https://metrics.hpc.arizona.edu/#job_viewer?action=show&realm=SUPREMM&resource_id=73&local_job_id=5909957 by 8am on 2022/12/16.

✅ There is also a log, array_slurm_my_subjects.log, which summarizes the job number for each subject:

cat array_my_subjects.log
JOBNAME=array_my_subjects, JOB ID: 5909958, Array Index: 2, Subject: sub-rudolph
JOBNAME=array_my_subjects, JOB ID: 5909959, Array Index: 3, Subject: sub-fred
JOBNAME=array_my_subjects, JOB ID: 5909957, Array Index: 1, Subject: sub-014
JOBNAME=array_my_subjects, JOB ID: 5909960, Array Index: 4, Subject: sub-happy
JOBNAME=array_my_subjects, JOB ID: 5909956, Array Index: 5, Subject: sub-fluffy

That’s it. You’ve run an array job and examined the output!

Next Steps

Someday, in the future, you may wish to run array jobs on your own datasets. You'll see the sorts of scripts you might wish to modify in several upcoming lessons. Here are some basic instructions for how to set that up:

• To use your own script with sbatchr, add the following chunk of code to your script after the SLURM setup and before the code your script calls (just the way it is set up in array.sh).

############################################################
#### BEGIN STUFF TO ADD TO ARRAY SCRIPT CALLED BY SBATCHR ####

### SLURM can specify an informative name for the output: Jobname-Jobnumber.out
#SBATCH --output=%x-%a.out

### SLURM Inherits your environment. cd $SLURM_SUBMIT_DIR not needed

### This is critical for any call to sbatchr. It gets the subject names from the list.
### Note that the subject list can be specified, but defaults to subjects.txt in the current directory if it is not specified.
### Pull filename from line number = SLURM_ARRAY_TASK_ID
Subject="$( sed "${SLURM_ARRAY_TASK_ID}q;d" "${SUBJS:-subjects.txt}")"

### The following is useful for documentation purposes.
### The array index and subject number get echoed to every output file produced by sbatch.
### Print job information to each output job
pwd; hostname; date
loginfo="JOBNAME=$SLURM_JOB_NAME, JOB ID: $SLURM_JOB_ID, Array Index: ${SLURM_ARRAY_TASK_ID}, Subject: sub-${Subject}"

### Also create a log file for the job that echos the subject number and index of each subject job to a single log file.
echo ${loginfo} >>${SLURM_JOB_NAME}.log
echo ${loginfo}

#### END STUFF TO ADD TO ARRAY SCRIPT CALLED BY SBATCHR ####
############################################################

For running BIDS apps, the subject list should just include the subject value and not the key (e.g., 101 rather than sub-101)

Summary

• Most importantly, in this lesson you learned to use a batch script, and the more sophisticated batch array script. The batch array is a better choice if you have lots of subjects to process.
• Less critically, you learned:
  • Batch jobs require you to estimate the CPU time needed for each job. You'll find the scripts in /groups/dkp/neuroimaging/scripts have good estimates of how long each participant will take. You may find you need more resources if you have higher resolution data or more data per subject, but at least you have a starting point.
  • It is best to overestimate at first, so jobs are not killed unfinished. But if you overestimate too much, your jobs will be queued for a very long time!
  • sbatchr creates a SLURM array job for each subject. Because each subject is a separate job, all you need to know is what CPU and time resources one subject job requires. Whenever there is room in the HPC job queue to run one or more of your subject jobs, those jobs will start.
• There are other similar solutions out there.  

Topics

• Batch Job 

• Batch Array Job 

• cluster 

• SLURM

Required

✅ Download sub-CAM003_ses-01brain_brain.nii.gz from the derivatives directory and view it in Mango on your own computer.

✅ Delete the derivatives directory from Practice 1. You will create a new derivatives directory when you run other BIDS apps in the future.

Encouraged

 In the Discussion: HPC

✅ Propose a topic based on what was difficult or what you would like to know more about.

✅ Did you find a resource (e.g., website, video, article) that was helpful? Share the link and tell us why you liked it.

✅ Read your classmates' posts and provide feedback.