Taxonomy

Overview

The taxonomy tool is part of BBTools' comprehensive taxonomy package designed for processing NCBI taxonomy information. It provides taxonomic classification for sequences and organisms using various identifier formats, working in conjunction with other BBTools taxonomy utilities to enable sequence filtering, binning, and classification workflows.

Database Setup

Before using taxonomy.sh, you must set up NCBI taxonomy databases. The simplest approach uses the automated pipeline script:

Automated Setup (Recommended)

# Navigate to desired taxonomy directory
cd /usr/tax/

# Run automated fetch and format script
fetchTaxonomy.sh

# Use with taxonomy tools
taxonomy.sh taxpath=/usr/tax/ homo_sapiens

Manual Setup (Advanced Users)

# Download taxonomy data from NCBI
wget ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdmp.zip
wget ftp://ftp.ncbi.nih.gov/pub/taxonomy/accession2taxid/*.gz

# Extract and process taxonomy files
unzip taxdmp.zip
taxtree.sh names.dmp nodes.dmp tree.taxtree.gz

# Optional: Create gi table for gi number support
gitable.sh shrunk.*.accession2taxid.gz gitable.int1d.gz

Supported Sequence Formats

The taxonomy tool recognizes six standard sequence header formats. Understanding these formats is crucial for proper taxonomic identification:

1. NCBI Taxonomic Identifier (Recommended)

>tid|123|other stuff

Advantage: Fastest processing, lowest memory usage, no lookup tables required.

2. GI Number (Legacy NCBI)

>gi|123|other stuff

Requirements: Requires gitable.int1d.gz file (~4GB RAM).

3. Accession Number (Current NCBI)

>AC1234.1 other stuff

Requirements: Requires accession2taxid files (~45GB RAM).

4. Latin Scientific Names

>Homo_sapiens
>Escherichia_coli
>Gammaproteobacteria

Format: Genus_species or higher taxonomic levels. Case-insensitive, underscores optional.

5. Silva Database Format

>GCVF01000431.1.2369 Bacteria;Proteobacteria;Gammaproteobacteria;...

Requirements: Use silva=t parameter. Semicolon-delimited taxonomy follows accession.

6. IMG Database Format

>2547132486 EuzAGGAGA464_NODE_7_len_49232_cov_91_7699...
>img|2547132486| other stuff
>tid|123|img|2547132486| other stuff

Usage: For IMG genome data, often used with renameimg.sh preprocessing.

Basic Usage

taxonomy.sh tree=<tree file> <identifier>
taxonomy.sh tree=<tree file> in=<file>

The taxonomy tool accepts organism identifiers as command-line arguments or reads them from input files. For users at NERSC/Genepool, use tree=auto to access pre-built databases.

Parameters

Examples

taxonomy.sh tree=tree.taxtree.gz homo_sapiens canis_lupus 9606

taxonomy.sh tree=tree.taxtree.gz gi=gitable.int1d.gz in=sequences.fasta out=taxonomy_results.txt

taxonomy.sh tree=tree.taxtree.gz level=genus in=species_list.txt

taxonomy.sh tree=tree.taxtree.gz silva=t in=silva_sequences.fasta

taxonomy.sh tree=tree.taxtree.gz minlevel=family maxlevel=kingdom in=organisms.txt

# Setup taxonomy databases
cd /path/to/taxonomy/
fetchTaxonomy.sh

# Use with custom taxonomy path
taxonomy.sh taxpath=/path/to/taxonomy/ tree=auto gi=auto escherichia_coli

# Instead of local databases, use web service
gi2taxid.sh in=sequences.fa out=renamed.fa server

# Or query directly via web browser
# https://taxonomy.jgi.doe.gov/name/homo_sapiens,escherichia_coli
# https://taxonomy.jgi.doe.gov/id/9606
# https://taxonomy.jgi.doe.gov/gi/123456

Algorithm Details

The taxonomy tool implements high-performance taxonomic classification using PrintTaxonomy.java with optimized TaxTree data structures and multi-stage identifier resolution.

Core Processing Architecture

PrintTaxonomy orchestrates taxonomic lookup through specialized processing pipelines:

• processNames(): Command-line identifier processing with direct taxLevelExtended comparison for immediate results
• processFile(): TextFile line-by-line processing with optional keyColumn extraction for structured data
• processReads(): ConcurrentReadInputStream processing enabling multi-threaded FASTA/FASTQ sequence handling
• parseNodeFromHeader(): Delegates to TaxTree.parseNodeFromHeader() for identifier pattern recognition and extraction

TaxTree Data Structure Implementation

The TaxTree class maintains NCBI's complete taxonomic hierarchy using memory-optimized storage:

• nodes array: Direct O(1) access to TaxNode objects indexed by taxonomic ID, enabling instant lookups
• nameMap HashMap: Scientific name to node list mapping with collision handling for synonyms and alternative names
• getNodesByNameExtended(): Returns List<TaxNode> supporting multiple matches for ambiguous scientific names
• levelExtended encoding: Numeric hierarchy levels enabling efficient parent-child traversal and level filtering

Multi-Stage Identifier Resolution

Three specialized mapping systems handle different NCBI identifier formats with optimized storage:

• GiToTaxid.initialize(): Integer array mapping from gi numbers to taxIDs, requires ~4GB RAM for complete NCBI gi database
• AccessionToTaxid.load(): HashMap-based string mapping from accessions to taxIDs, requires ~45GB RAM for complete NCBI accession database
• TaxTree.parseNodeFromHeader(): Regular expression pattern matching for embedded identifiers in sequence headers across multiple formats
• Direct taxID resolution: Array index access via tree.getNode(taxID) provides fastest possible lookup performance

Tree Traversal and Lineage Generation

Taxonomic lineage construction uses parent-following traversal with intelligent filtering:

• Parent traversal loop: while(tn.id != tn.pid) using tn=tree.getNode(tn.pid) follows parent links to root
• Level-based filtering: tn.levelExtended comparisons against minLevelExtended/maxLevelExtended parameters enable taxonomic range selection
• Canonical rank filtering: tn.isSimple() method excludes non-standard taxonomic ranks like 'no rank' and intermediate levels
• Output formatting: TextStreamWriter with configurable tab-delimited level/id/name structure or semicolon-delimited format

Output Generation Strategies

Multiple output methods accommodate different analytical workflows:

• printTaxonomy(): Complete lineage generation with tab-delimited level/id/name format for comprehensive taxonomic information
• printTaxLevel(): Single taxonomic level extraction using level comparison and parent traversal until target level reached
• makeTaxLine(): Semicolon-delimited format with level__name structure compatible with phylogenetic analysis tools
• translateLine(): In-place column replacement for tab-delimited files, enabling taxonomy annotation of existing datasets

Memory Management and Scalability

Resource usage scales with database components loaded:

• TaxTree only: NCBI nodes array with HashMap requires 8-16GB RAM, loads in 2-5 seconds
• With GiToTaxid: Additional 2-4GB memory overhead for gi number mapping, extends loading to ~1 minute
• With AccessionToTaxid: String key HashMap storage requires ~45GB for complete NCBI accession data, loading takes 5-10 minutes
• Processing throughput: Limited by identifier resolution speed rather than tree traversal, which operates at array access speeds

Concurrency and Thread Safety

Multi-threaded processing components enable scalable sequence analysis:

• ConcurrentReadInputStream: Multi-threaded FASTA/FASTQ parsing with configurable thread pools for high-throughput processing
• TextStreamWriter: Thread-safe output stream management with internal buffering and synchronization
• Immutable TaxTree: Read-only tree structure after loading supports unlimited concurrent access without synchronization overhead
• Thread-safe mappings: GiToTaxid and AccessionToTaxid provide concurrent lookup capability across processing threads

Taxonomy Database Management

Understanding taxonomy database requirements and setup procedures is essential for effective use of the taxonomy tool.

Database Files Overview

NCBI Data Sources

# Core taxonomy data (always required)
ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdmp.zip

# Accession mappings (optional, for accession format sequences)
ftp://ftp.ncbi.nih.gov/pub/taxonomy/accession2taxid/nucl_gb.accession2taxid.gz
ftp://ftp.ncbi.nih.gov/pub/taxonomy/accession2taxid/nucl_wgs.accession2taxid.gz
ftp://ftp.ncbi.nih.gov/pub/taxonomy/accession2taxid/prot.accession2taxid.gz
# ... additional accession2taxid files as needed

Database Generation Commands

# Extract and process core taxonomy
unzip taxdmp.zip
taxtree.sh names.dmp nodes.dmp tree.taxtree.gz

# Generate accession mappings (if needed)
gitable.sh nucl_gb.accession2taxid.gz,nucl_wgs.accession2taxid.gz gitable.int1d.gz

Sequence Optimization Workflow

To minimize memory requirements and maximize performance, convert sequences to taxID format:

# Convert gi/accession sequences to taxID format (recommended)
gi2taxid.sh in=sequences.fasta out=renamed.fasta tree=auto table=auto accession=auto taxpath=/path/to/taxonomy/

# Now use taxonomy tool with minimal memory
taxonomy.sh tree=tree.taxtree.gz in=renamed.fasta out=results.txt

Database Location Strategies

• NERSC/Genepool users: Use tree=auto, gi=auto, accession=auto parameters
• Other users: Specify full paths or use taxpath parameter for database directory
• Shared systems: Install databases in /usr/local/tax/ or similar shared location
• Personal use: Create taxonomy directory in home directory or project space

Integration with BBTools Ecosystem

The taxonomy tool serves as a foundation for BBTools' comprehensive taxonomic analysis capabilities.

Preprocessing Pipeline

# 1. Setup taxonomy databases
fetchTaxonomy.sh

# 2. Convert sequences to efficient taxID format
gi2taxid.sh in=sequences.fa out=renamed.fa taxpath=/taxonomy/path/

# 3. Get taxonomic classifications
taxonomy.sh tree=auto in=renamed.fa out=classifications.txt taxpath=/taxonomy/path/

Taxonomic Filtering Workflow

# Filter sequences by taxonomy
filterbytaxa.sh in=sequences.fa out=bacteria_only.fa names=Bacteria level=superkingdom include=t tree=auto

# Split sequences into taxonomic bins
splitbytaxa.sh in=sequences.fa out=%.fa level=phylum tree=auto

Abundance Analysis Integration

# Generate taxonomic counts
taxonomy.sh tree=auto in=sequences.fa counts=taxa_counts.txt

# Classify reads with Seal at specific taxonomic level
seal.sh in=reads.fq ref=database.fa out=classified.fq taxonomy level=genus tree=auto

Quality Control and Validation

• Unknown classifications: Monitor "UNKNOWN" entries in output for sequence header parsing issues
• Memory monitoring: Use -Xmx parameter tuning based on database components loaded
• Performance optimization: Convert to taxID format for repeated analyses to avoid lookup table loading
• Cross-validation: Compare results with JGI taxonomy server for critical analyses

Support and Resources

Comprehensive support resources for taxonomy tool usage and troubleshooting.

Documentation

• Primary Guide: bbtools/docs/guides/TaxonomyGuide.txt - Comprehensive taxonomy workflow documentation
• Website: bbmap.org - Latest documentation and updates
• Alternative Service: taxonomy.jgi.doe.gov - Web-based taxonomy queries

Contact and Community

• Email Support: bbushnell@lbl.gov
• Bug Reports: GitHub issues or direct email to developer

Performance Optimization Tips

• Use taxID format sequences (tid|123|) for best performance and lowest memory usage
• Load only required database components (avoid accessions if not needed)
• Consider JGI taxonomy server for occasional queries instead of local database setup
• Use fetchTaxonomy.sh for simplified database setup and maintenance