BBDuk

Basic Usage

bbduk.sh in=<input file> out=<output file> ref=<contaminant files>

Input may be stdin or a fasta or fastq file, compressed or uncompressed. If you pipe via stdin/stdout, please include the file type; e.g. for gzipped fasta input, set in=stdin.fa.gz

Memory Management

BBDuk autodetects available memory and uses approximately half by default. Memory requirements are linearly proportional to reference size: only processing large references, or using high values of hdist or edist, require substantial memory. E. coli (4.6MB) uses ~140MB with default settings; hdist=1 expands to ~15GB due to storing 94× more kmers (427 million vs 4.5 million).

Threading Architecture

Processing occurs in two phases: Phase 1 loads reference kmers using 7 fixed threads (typically under 1 second for small references). Phase 2 autodetects and uses all available processors unless restricted with the 't' parameter. BBDuk scales well across multiple cores unless on a shared node.

Common Use Cases

Typical Preprocessing Pipeline Order

For complete read preprocessing, the recommended multi-pass approach follows this order:

1. Force trim modulo: ftm=5 removes inaccurate extra bases (151bp → 150bp)
2. Adapter trimming: ktrim=r k=23 mink=11 hdist=1 tpe tbo
3. Quality trimming: qtrim=r trimq=10 (uses Phred algorithm)
4. Contaminant filtering: Remove PhiX, rRNA, or custom references
5. Length/quality filtering: minlen=50 maq=20

Built-in References

BBDuk includes commonly-used reference sequences accessible by keyword:

• adapters: Illumina TruSeq and Nextera adapter sequences
• phix, lambda, pjet, mtst, kapa: Common sequencing controls and contaminants
• artifacts: Known sequencing artifacts from the BBTools resources

Parameters

BBDuk parameters are organized by function. Note: if ktrim, kmask, and ksplit are unset, the default behavior is kfilter. All kmer processing modes are mutually exclusive. Reads only get sent to 'outm' based on kmer matches in kfilter mode.

Examples

# High-quality data (recommended settings)
bbduk.sh in=reads.fq out=clean.fq ref=adapters ktrim=r k=23 mink=11 hdist=1 tpe tbo

# Low-quality data (more sensitive settings)
bbduk.sh in=reads.fq out=clean.fq ref=adapters ktrim=r k=21 mink=11 hdist=2

# Quality trimming using Phred algorithm
bbduk.sh in=reads.fq out=clean.fq qtrim=r trimq=10

# Both ends quality trimming with length filtering
bbduk.sh in=reads.fq out=clean.fq qtrim=rl trimq=10 minlen=50

# Remove inaccurate extra base (151bp → 150bp)
bbduk.sh in=reads.fq out=clean.fq ftm=5

# Trim specific positions
bbduk.sh in=reads.fq out=clean.fq ftl=10 ftr=139

# Remove PhiX contamination with statistics
bbduk.sh in=reads.fq out=unmatched.fq outm=matched.fq ref=phix k=31 hdist=1 stats=stats.txt

# Multi-reference filtering using built-in keywords
bbduk.sh in=reads.fq out=clean.fq ref=adapters,phix,artifacts k=27

# Full preprocessing following recommended order
bbduk.sh in1=raw_R1.fastq in2=raw_R2.fastq \
         out1=clean_R1.fastq out2=clean_R2.fastq \
         outm1=contaminated_R1.fastq outm2=contaminated_R2.fastq \
         ref=adapters,phix ktrim=r k=27 mink=11 hdist=1 \
         qtrim=rl trimq=10 ftm=5 minlen=50 maq=20 \
         tpe tbo stats=contamination_stats.txt

# RNA-seq with poly-A trimming and rRNA removal
bbduk.sh in=rnaseq.fastq out=clean.fastq ref=ribokmers.fa.gz \
         k=31 trimpolya=10 minlen=25 qtrim=r trimq=8

# Small RNA sequencing
bbduk.sh in=smallrna.fastq out=clean.fastq ref=adapters \
         ktrim=r k=19 mink=8 mininsert=16 minlen=16

# Use query distance for large references with limited memory
bbduk.sh -Xmx8g in=reads.fq out=clean.fq ref=large_genome.fa qhdist=1 k=31

# Reduce memory usage with skip patterns
bbduk.sh in=reads.fq out=clean.fq ref=large_genome.fa rskip=3 k=27

# Three-way splitting: pass pairs, fail pairs, pass singletons
bbduk.sh in=pairs.fq out=pass_pairs.fq outm=fail_pairs.fq \
         outs=pass_singletons.fq ref=contam.fasta

# Generate multiple histograms during processing
bbduk.sh in=reads.fq out=clean.fq bhist=base_hist.txt \
         qhist=qual_hist.txt gchist=gc_hist.txt lhist=len_hist.txt

Algorithm Details

Kmer Processing Modes

BBDuk operates in one of four mutually exclusive kmer modes (only one kmer-based operation per pass):

• kfilter (default): Separate reads based on kmer matches - reads matching reference go to outm
• ktrim=r/l: Remove matching kmer and all bases to right/left, keeping the unmatched portion
• kmask: Replace matching bases with N or lowercase without removing sequence
• ksplit: Split single-ended reads around kmer matches (pairs go to outm, singletons to out)

Kmer Table Implementation

BBDuk uses AbstractKmerTable data structures for kmer storage and lookup, with automatic selection based on reference characteristics:

• Array-based (default): Direct indexing for fast lookups, suitable for most references
• Hash table-based: Alternative implementation when memory becomes critical
• Multi-way partitioning: WAYS=7 hash-based distribution across processing threads for load balancing
• Memory scaling: Approximately 20 bytes per stored kmer; initialSize calculation based on available memory

Edit Distance Processing

Hamming and edit distance processing expands the kmer space through systematic mutation enumeration:

• Hamming distance (hdist): Substitutions only, complexity 1+(3×K)^hdist
• Edit distance (edist): Substitutions and indels, complexity 1+(8×K)^edist
• Memory example: E. coli k=31 hdist=0 stores 4.5M kmers (140MB), hdist=1 stores 428M kmers (15GB)
• Query distance alternative: qhdist/qedist mutate read kmers instead, constant memory but reduced speed
• Short kmer controls: hdist2/edist2 provide independent distance settings for mink-generated kmers

Sensitivity Enhancement

Multiple features increase matching sensitivity in the presence of sequencing errors:

• Middle masking (maskmiddle): Treat central base(s) as wildcard using configurable bitmask
• Variable kmer length (mink): Use shorter kmers at read ends when full-length kmers unavailable
• Reverse complement (rcomp): Match both forward and reverse-complement orientations
• Kmer hits requirements: Adjustable minkmerhits, minkmerfraction, mincovfraction thresholds

Performance Optimization Strategies

BBDuk provides multiple acceleration options, though all reduce sensitivity as a trade-off:

• Kmer space sampling (speed): Ignore speed/16 fraction of kmers in both reads and reference
• Reference skipping (rskip): Sample reference kmers to reduce memory and loading time
• Query skipping (qskip): Sample read kmers to increase processing speed
• Threading model: Two-phase processing with 7 fixed loader threads, then all available processors
• Important: Avoid combining speed, qskip, and rskip flags - use only one acceleration method

Internal Order of Operations

BBDuk processes operations in this fixed sequence (reads discarded as soon as length drops below minlen):

1. Histograms and cardinality estimation (on raw reads)
2. Header filtering (chastity, location, barcode)
3. Quality recalibration
4. GC filtering
5. Force trimming (ftl/ftr/ftm)
6. Kmer operations (ktrim/kmask/kfilter - mutually exclusive)
7. Trim by overlap (tbo)
8. Poly-A trimming
9. Entropy masking
10. Quality trimming (qtrim) and soft-clip trimming
11. Average quality filtering (maq) and maxNs filtering
12. Entropy filtering

Workflow Integration

Paired Read Processing

BBDuk automatically detects paired reads based on read names and maintains pair relationships throughout processing:

• Input formats: Single interleaved file (autodetected) or separate files (in1/in2)
• Pair coordination: Both reads processed together - pairs kept together or both discarded
• rieb parameter: Controls whether both reads must fail (rieb=f) or either read failing removes the pair (rieb=t, default)
• Output flexibility: out/out2 (passed pairs), outm/outm2 (failed pairs), outs (passed singletons when mate fails)

Multi-Reference Processing

When using multiple reference files, BBDuk associates each kmer with the first reference containing it:

• Kmer ownership: Each kmer belongs to only one reference (the first encountered)
• Statistics impact: Shared kmers between references affect per-reference counting accuracy
• Alternative tool: Use Seal instead of BBDuk when accurate per-reference statistics are critical with overlapping sequences
• Built-in references: Keywords (adapters, phix, etc.) map to curated sequences in BBTools resources

Pipeline Integration

BBDuk serves as a central preprocessing tool in typical sequencing workflows:

• Input diversity: Handles FASTA, FASTQ, compressed formats, streaming input/output
• Quality workflows: Integrates with BBMap alignment, CalcTrueQuality recalibration, assembly pipelines
• Output coordination: Multiple output streams support complex downstream processing requirements
• Resource management: Automatic memory detection and thread scaling simplify cluster deployment

Support

For questions and support:

• Email: bbushnell@lbl.gov
• Documentation: bbmap.org
• Guide: Read bbtools/docs/guides/BBDukGuide.txt for detailed information and additional examples