Working with a HPC Job Scheduler
In this workshop, we have been running each step of fastqc, trimming, indexing, and aligning directly on the command line. This is not always best practise, for a few reasons, and what you will likely need to do is translate this code into a script. For one, it is easier to reproduce and automate your workflow when everything you do is in a script. But perhaps more importantly, when working on a HPC environment, your direct command line inputs are operating on the head node or log in node. Many of these software used require more CPUs/memory than can be allocated on the head node and therefore you need to submit your job to the HPC job scheduler (e.g., SLURM or PBS) to run the job more efficiently on allocated compute resources.
See here our GA workshop on Bash Scripting and HPC Job Scheduler for more details on how to prepare, run and optimise a SLURM script.
Example only: Submitting STAR alignment to SLURM
Once we have confirmed the STAR alignment code works for our first sample, the next step would be to align all the samples to the genome. As this would take quite a while, we won’t do that in this workshop and we will instead supply the outputs for all samples. We could do all the alignments with a for loop, but as we have many samples and for loops run one sample at a time, this is not the most efficient way to do this. We need to write a slurm script as an array job and submit it to the HPC job scheduler (i.e., SLURM on REANNZ) to allocate CPUs and memory for the job. We won’t be doing this in this workshop either, but you can see an example of a SLURM script below.
First, make a samples.txt file that contains the basename of the files. This way we can extract the prefix we want for the outfiles in a way that works nicely with an array job.
ls path/to/Trimmed/*_R1_trimmed.fq.gz | xargs -n1 basename | sed 's/_R1_trimmed\.fq\.gz//' > samples.txtNote: this script assume you are currently in your alignment dir (where you want the star alignment results to go), which is where the samples.txt file will also go, and that your trimmed read files are in a directory called ../Trimmed relative to your current directory.
As an example, this below is what a slurm script may look like for running the STAR align step. Note the use of things like relative paths – you need to adjust the paths to work from the directory in which your script is submitted from.
Slurm scripts are text files, and use the .sl extension by convention. You can use nano to create a script.
nano star-align.sl Then paste in the below text into nano, save and exit. You may need to adjust the amount of memory, time and CPUs allocated for the job depending on the size of your data. As this is set up as an array job, each sample will be aligned in parallel in its own job (each with the amount of memory and CPUs specified), which is much more efficient than running a for loop.
#!/bin/bash
#SBATCH --job-name=star_align_array
#SBATCH --array=1-15 # adjust to number of samples you have
#SBATCH -A nesiprojectID
#SBATCH --output=logs/star_%A_%a.out
#SBATCH --error=logs/star_%A_%a.err
#SBATCH --mem=20G
#SBATCH --cpus-per-task=8
#SBATCH --time=05:00:00
module load STAR/2.7.10b-GCC-11.3.0-alpha # current module version on REANNZ
# Set dir paths
TRIMMEDDIR=/path/to/trimmed # contains your trimmed reads
OUTDIR=/path/to/star_out # dir where you want your output files to go
GENOMEDIR=/path/to/genome-files # contains your indexed genome
# Make logs dir for slurm out and err files
mkdir -p $OUTDIR logs
# Set shell variables
sample=$(sed -n "${SLURM_ARRAY_TASK_ID}p" samples.txt)
r1=$TRIMMEDDIR/${sample}_R1_trimmed.fq.gz
r2=$TRIMMEDDIR/${sample}_R2_trimmed.fq.gz
STAR \
--runMode alignReads \
--runThreadN $SLURM_CPUS_PER_TASK \
--genomeDir $GENOMEDIR \
--readFilesIn $r1 $r2 \
--readFilesCommand zcat \
--outSAMtype BAM SortedByCoordinate \
--outFileNamePrefix $OUTDIR/${sample}_ \
--quantMode GeneCounts Now submit the following script to SLURM to align each sample to the genome.
To submit the script, from your working directory you would type:
sbatch star-align.sl To monitor the progress of the job, you would type:
squeue --meand/or
sacct