Sample pre-processing for UKBOL accelerated

SOP for pre-processing of raw genome skims generated from museum specimens, going from sample metadata and non-base called PE short read NGS data to recovering barcoding gene(s) and mitogenome assembly. For the UKBOL accelerated project @NHMUK.

Workflow

1. Generating sample taxonomy file
2. Basecalling and demultiplexing AVITI sequence data
3. Preparing sequence data
4. Merging lane data
5. Preparing the samples sheet
6. Recovering and validating barcodes with BeeGees
7. Assembling and validating mitochondrial genomes with (modified) skim2mito

1. Generating sample taxonomy file

This section describes the process of generating the necessary sample metadata file from specimen metadata.

1. The bioinf_taxonomy sheet in the specimen sampling UK*_processing.xlsx will be automatically populated when filling out the ‘Processing’ sheet of the same UK*_processing.xlsx file.
2. Once the specimen sampling spreadsheet is completed, ensure you are ‘on’ the bioinf_taxonomy sheet, click File, Export, and Download CSV as UTF-8 (encoding). This will automatically save the bioinf_taxonomy sheet to your Downloads/ folder as UK*_processing(bioinf_taxonomy).csv.
3. Rename and copy this newly generated UK*_processing.csv onto the appropriate area of the cluster (e.g. ~/projects/nhm/museomix/UKBOL_accelerated/sample_metadata/), and update the batch tracking spreadsheet accordingly.
4. Perform a sanitisation check of UK*_processing.csv (sanitisation check = Manually check ID and phylum -> species columns for special characters, trailing spaces, tabs, etc and remove them).

The UK*_processing.csv will contain the following fields:

Sample ID	phylum	class	order	family	genus	species
UK001-A01	Arthropoda	Insecta	Coleoptera	Carabidae	Carabus	speciousus

Sample ID = Sample unique identifier (NO underscore characters)

2. Basecalling and demultiplexing AVITI sequence data

This section describes the process of basecalling and demultiplexing raw data from the AVITI using bases2fastq on the NHM HPC. Raw AVITI data is written to a folder on NHM HPC with restricted access (/mbl/share/instruments/aviti/AV251604/).

Setup and installation (only need to do steps 1-3 once):

1. Create conda environment and install container software singularity (Singularity is a container platform (now called Apptainer). It ensures software runs consistently by packaging all dependencies. Either singularity or apptainer command works on the museum HPC).

   conda create -n singularity
   conda activate singularity
   conda install -c conda-forge singularity

2. Download bases2fastq container and rename with version

   singularity pull docker://elembio/bases2fastq
   mv bases2fastq_latest.sif bases2fastq_2.3.0.sif
   

Basecalling and demultiplexing:

3. Edit the following two paths for each new run in the bases2fastq.slurm script

INPUT_DIR="/mbl/share/instruments/aviti/AV251604/RUNFOLDER"
OUTPUT_DIR="/hpc/groups/genomics-collections/ukbol_accelerated/aviti_data/RUNFOLDER_ANALYSIS"

(Adding “_ANALYSIS” to the runfolder for the output directory name distinguishes the basecalled folder)

4. Submit job

sbatch bases2fastq.slurm

The bases2fastq runtime is typically ~1 hour for a 2x75 run but will vary based on sequencing run length and HPC system load/job queue length.

3. Preparing the sequence data

This section outlines the transfer of sequence data from the NHM to the Crop Diversity cluster (if desired), and the process of generating the samples sheet file required to run the BeeGees and skim2mito pipelines.

1. First, compress the sequence data directory for faster transfer using compress.sh. Change the input directory path in the script. Requires pigz.
2. Second, generate an md5 checksum for the compressed archive, as follows:

md5sum [COMPRESSED_SEQ_DATA].tar.gz > [COMPRESSED_SEQ_DATA].tar.gz.md5

3. Next, transfer sequence data (along with the md5 file) from the NHM cluster to the Crop Diversity cluster using rsync:

rsync -avzP /PATH/TO/[COMPRESSED_SEQ_DATA].tar.gz.* [USERNAME]@gruffalo.cropdiversity.ac.uk:/PATH/TO/OUTPUT/LOCATION

  -a: Copy recursively, maintaining permissions and timestamps
  -v: verbose logging and progress tracking
  -P: Allows user to resume transfer if it interrupted (by rerunning the same command)

• Remember to include the wildcard on the compressed sequence data directory to also transfer the md5 checksum

4. Once the transfer is complete, verify the MD5 checksum is correct (on the Crop Diversity cluster) using:

md5sum -c [COMPRESSED_SEQ_DATA].tar.gz.md5

4. Merging lane data

This section describes the merging of lane1 and lane2 sequencing data.

1. In the crop diversity cluster, rename the runfolder with the batch name (e.g. UK001).

2. Navigate to the Samples folder in the run folder. This contains the raw data from the separate lanes.

   cd [BATCHNAME]/Samples/

3. Copy the mergelanesV2.sh script into this directory and run the script.

   ./mergelanesV2.sh

4. The output folder containing the merged reads (one R1 and R2 file per sample) will be named merged_samples. Move this folder to this directory ~/projects/nhm/museomix/UKBOL_accelerated/aviti_data/merged_samples and rename it to the batch name (e.g. UK001).

5. Preparing the samples sheet

This section outlines the process of generating the samples sheet (needed for downstream pipelines) from the already rpepared sample metadata (UK*_processing.csv) files. It consists of two main steps; (1) generating the preliminary samples sheet with sample_processing.py, and (2) retrieving the cloest available NCBI taxonomic identifier (taxid) using each sample’s hierarchical taxonomic information via taxid_fetcher.py.

1. Run sample_processing.py to generate the samples sheet, and save it to ~/projects/nhm/museomix/UKBOL_accelerated/sample_sheets/

  python sample_processing.py path/to/[seq_data_parent_directory] path/to/sample_metadata [output_directory] --identifier "Sample ID"

Required arguments:
seq_data_parent_directory: Directory containing FASTQ/FQ files (Flat directory format, or nested)
sample_metadta: CSV file with sample metadata (e.g. bioinf_taxonomy.csv)
output_directory: Directory where output files will be saved (will be created if it doesn't exist) (e.g. ~/projects/nhm/museomix/UKBOL_accelerated/sample_sheets)
--identifier/--id: Column name in sample_metadata CSV containing sample IDs (default: "Process ID")

Additional optional arguments:
--output-prefix [output_prefix]: Prefix for output files (defaults to input parent directory name)
--merge: Prefer '*_merged' directories when available, fall back to non-suffixed directories

2. Manually check generated samples sheet paths and taxonomic data parsing proceeded correctly

The generated [samples].csv will contain the following fields:

ID	forward	reverse	phylum	class	order	family	genus	species
UK001-A01	abs/path/R1.fastq.gz	abs/path/R2.fastq.gz	Arthropoda	Insecta	Coleoptera	Carabidae	Carabus	speciousus

2. Run taxid_resolver.py to retrieve the closest valid (i.e. available) NCBI taxonomic identifier (taxid) for each sample in the samples sheet using the sample taxonomic lineage (phylum->species) information:

  python taxid_resolver.py --input [samples sheet] --email [NCBI email address] --api [NCBI API key]

NCBI API credentials can be created by following the instructions described here.

• The script will retrieve and append the taxid to the input samples sheet, as well as matched_rank and lineage_mismatch columns.

matched_rank: Contains the rank at which a valid taxonomic identifier could be retrieved (e.g. species)
lineage_mismatch: Boolean (YES/NO) variable highlgihting whether the input phylum and/or class ranks differed to the phylum and/or class ranks of the lineage for the returned taxid. 'YES' might reflect a taxonomic homonym.

3. Manually check output samples sheet to confirm lineage_mismatch == NO for all samples

6. Recovering and validating barcodes with BeeGees

• Requirements at a glance (see BeeGees repo for more details):
  • Cloned BeeGees repo
  • Populated samples sheet
  • Conda environment with neccessary dependencies installed
  • Modified config.yaml for run

7. Assembling and validating mitochondrial genomes with (modified) skim2mito

• Requirements at a glance (see skim2mito repo for more details):
  • Cloned skim2mito repo
  • Populated samples sheet
  • Conda environment with neccessary dependencies installed
  • Modified config.yaml for run

Contributions and authorship

This SOP was designed, tested, and implemented by Dan Parsons & Jordan Beasley @NHMUK.