TestFormat2

Basic Usage

testformat2.sh <file>

The tool accepts a single file as input and performs analysis to determine file format, compression, quality metrics, base composition, and various other characteristics important for bioinformatics pipeline planning.

Parameters

TestFormat2 organizes parameters into functional groups for analysis control and output generation. The tool provides customization options for different analysis depths and output formats.

Examples

# Analyze a FASTQ file with default settings
testformat2.sh reads.fastq

# Quick format detection without full analysis
testformat2.sh reads.fastq full=f

# Comprehensive analysis with timing information
testformat2.sh reads.fastq speed=t

# Full analysis with custom histogram files
testformat2.sh reads.fastq qhist=quality_dist.txt ihist=insert_sizes.txt

# Barcode analysis with expected barcode validation
testformat2.sh barcoded_reads.fastq barcodelist=expected_barcodes.txt printbarcodes=t

# PacBio data analysis with ZMW tracking
testformat2.sh pacbio_reads.fastq zmw=t zmwhist=zmw_passes.txt

# Quality assessment with trim simulation
testformat2.sh reads.fastq trim=t printqhist=t

# Disable sketching for faster processing of large files
testformat2.sh large_file.fastq sketch=f merge=f

# Generate all histogram files for comprehensive QC
testformat2.sh reads.fastq hist=t bhistlen=0

Algorithm Details

Multi-threaded Analysis Architecture

TestFormat2 employs a sophisticated multi-threaded processing architecture that automatically scales thread usage based on system resources. For systems with 48+ threads, the tool intelligently limits usage to a maximum of 40 threads or half the available threads to prevent resource contention while maintaining optimal performance.

Format Detection Strategy

The tool uses a dual-phase format detection approach: filename-based initial detection followed by content-based validation. This strategy first examines file extensions and compression indicators, then validates the prediction by analyzing file headers and content structure. Quality offset detection specifically tests both Phred+33 and Phred+64 encodings to ensure accurate quality score interpretation.

Statistical Analysis Methods

Base composition analysis uses specialized lookup tables (toNum, toLUS, toAmino arrays) for high-speed character classification. Quality score statistics employ error probability calculations using QualityTools.PROB_ERROR arrays for accurate error rate estimation. Length statistics utilize SuperLongList data structures optimized for large datasets with efficient median and mode calculations.

Sketching and Cardinality Estimation

When sketching is enabled, the tool implements MinHash sketching through SketchMakerMini with configurable sketch sizes (default 40,000). The sketching algorithm automatically detects PacBio vs Illumina data characteristics and adjusts parameters accordingly. Cardinality estimation uses both simple and 3+ k-mer frequency thresholds for robust genome size estimation, with optional RefSeq server queries for taxonomic identification.

Merging and Insert Size Analysis

Paired-read merging employs the BBMerge.findOverlapLoose algorithm to detect read pair overlaps without requiring perfect matches. The algorithm identifies adapter contamination by comparing successful merge lengths with read lengths, tracking both adapter-containing reads and total adapter bases. Insert size histograms are generated from successful merges with automatic binning optimization.

Quality-Based Trimming Simulation

Trim analysis uses TrimRead.testOptimal with configurable quality thresholds (Q5, Q10, Q15, Q20) to simulate various trimming strategies. The algorithm employs error probability matrices to determine optimal trim points that balance data retention with quality requirements, providing predictive data for downstream processing decisions.

Memory Management and Scalability

The tool implements adaptive memory management with automatic heap size calculation based on available system memory (calcXmx function). Large file processing uses streaming analysis to minimize memory footprint while maintaining statistical accuracy. Histogram data structures are pre-allocated with appropriate sizes (256 quality bins, 1000 insert size bins) to prevent memory fragmentation during analysis.

Error Handling and Data Validation

Robust error detection identifies various data quality issues including invalid base characters, malformed quality strings, mismatched pair names, and chastity filter failures. The tool maintains separate counters for different error types, enabling detailed quality assessment and troubleshooting of problematic datasets.

Support

For questions and support:

• Email: bbushnell@lbl.gov
• Documentation: bbmap.org