TestFormat2

Basic Usage

testformat2.sh <file>

The tool accepts one or more files as input and performs analysis to determine file format, compression, quality metrics, base composition, and various other characteristics important for bioinformatics pipeline planning. Sequence statistics from multiple files are accumulated and printed once; GFF, BED, and VCF validation reports are printed per file.

If a validation or I/O error is detected, the tool prints its results and then exits with a nonzero status. Running with full=f skips full-file analysis and validation entirely.

Parameters

TestFormat2 organizes parameters into functional groups for analysis control and output generation. The tool provides customization options for different analysis depths and output formats.

Format Validation

Full-file analysis (full=t, the default) validates three annotation and variant formats instead of computing sequence statistics. Validation is limited to the checks listed below; anything not listed is unchecked, and each report states its scope.

GFF

Checks nine nonempty fields, control characters, positive ordered coordinates, decimal score syntax, strand, and phase (phase is required for CDS features). Attributes, directives, ontology, feature relationships, and embedded FASTA contents are not validated. The report includes FeatureLines, InvalidFeatures, CommentLines, BlankLines, UncheckedFastaLines, and ValidationScope, plus the line number and message of the first error. If any core check fails, or a read, decompression, or close failure occurs, the tool exits nonzero.

BED

Checks standard fields and block structure, and accepts UCSC track headers. Custom columns beyond the standard field count are reported as unchecked. Reference bounds and sorting are not checked. Use bedfields (see Format Validation Parameters) to declare the standard field count for BEDn+ files; the default auto mode infers BED3-9 or BED12.

VCF

Checks VCF4.1 through VCF4.5 headers, column counts, POS, REF, and QUAL syntax, and FORMAT and sample column structure. Blank lines after the version line are tolerated; the final newline is not checked. The following remain unchecked: ALT and GT semantics, INFO, FILTER, and ID semantics, declaration matching, numeric ranges, reference agreement, and sorting.

VAR Summary Files

For VAR files the tool counts variant lines and header lines, and reads the "#" header keys ploidy, pairingRate, totalQualityAvg, mapqAvg, and readLengthAvg. Each value is printed only when its header is present. Variants per second is reported when speed=t.

Multiple Inputs and Mixed Types

Sequence statistics from several files are accumulated and printed once, using the metadata of the last file processed. GFF, BED, and VCF reports are per file, and their metadata does not carry into the sequence summary. In the current source, mixing sequence and VAR inputs suppresses the VAR summary; per-file reporting for mixed inputs is not implemented yet.

Examples

# Analyze a FASTQ file with default settings
testformat2.sh reads.fastq

# Header-level summary only, without full-file analysis
testformat2.sh reads.fastq full=f

# Analysis with timing information
testformat2.sh reads.fastq speed=t

# Full analysis with custom histogram files
testformat2.sh reads.fastq qhist=quality_dist.txt ihist=insert_sizes.txt

# Barcode analysis with expected barcode validation
testformat2.sh barcoded_reads.fastq barcodelist=expected_barcodes.txt printbarcodes=t

# PacBio data analysis with ZMW tracking
testformat2.sh pacbio_reads.fastq zmw=t zmwhist=zmw_passes.txt

# Quality assessment with trim simulation
testformat2.sh reads.fastq trim=t printqhist=t

# Disable sketching for faster processing of large files
testformat2.sh large_file.fastq sketch=f merge=f

# Keep the default histogram outputs and include reads of all lengths in bhist.txt
testformat2.sh reads.fastq hist=t bhistlen=0

# Validate a GFF3 file; exits nonzero if core checks fail
testformat2.sh annotation.gff3

# BED file with a standard BED6 prefix followed by custom columns
testformat2.sh regions.bed bedfields=6

# Validate a VCF file
testformat2.sh variants.vcf

Algorithm Details

Multi-threaded Analysis Architecture

TestFormat2 runs one worker thread per available thread reported by the runtime (Shared.threads()). When that count is 48 or more, the tool uses min(40, threads/2) workers to limit resource contention. Each worker processes lists of reads from a shared ConcurrentReadInputStream and keeps its own histograms, which are merged after all workers finish.

Format Detection Strategy

The tool uses a dual-phase format detection approach: filename-based initial detection followed by content-based validation. This strategy first examines file extensions and compression indicators, then validates the prediction by analyzing file headers and content structure. Quality offset detection specifically tests both Phred+33 and Phred+64 encodings to ensure accurate quality score interpretation.

Statistical Analysis Methods

Base composition analysis uses specialized lookup tables (toNum, toLUS, toAmino arrays) for high-speed character classification. Quality score statistics employ error probability calculations using QualityTools.PROB_ERROR arrays for accurate error rate estimation. Length statistics utilize SuperLongList data structures optimized for large datasets with efficient median and mode calculations.

Sketching and Cardinality Estimation

When sketching is enabled, the tool implements MinHash sketching through SketchMakerMini with configurable sketch sizes (default 40,000). The sketching algorithm automatically detects PacBio vs Illumina data characteristics and adjusts parameters accordingly. Cardinality estimation uses both simple and 3+ k-mer frequency thresholds for robust genome size estimation, with optional RefSeq server queries for taxonomic identification.

Merging and Insert Size Analysis

Paired-read merging employs the BBMerge.findOverlapLoose algorithm to detect read pair overlaps without requiring perfect matches. The algorithm identifies adapter contamination by comparing successful merge lengths with read lengths, tracking both adapter-containing reads and total adapter bases. Insert size histograms are generated from successful merges with automatic binning optimization.

Quality-Based Trimming Simulation

Trim analysis uses TrimRead.testOptimal for every quality threshold from Q5 through Q20, and reports the Q5, Q10, Q15, and Q20 results. The algorithm employs error probability matrices to determine optimal trim points that balance data retention with quality requirements, providing predictive data for downstream processing decisions.

Memory Management and Scalability

The launcher script passes --xmx=2g, --xms=2g, --percent=42, and --mode=auto to the shared Java argument parser, so heap size follows those defaults unless overridden. Large file processing uses streaming analysis to minimize memory footprint while maintaining statistical accuracy. Histogram arrays are fixed in size: 256 quality bins (offset by 128), 1000 insert size bins, and 51 trim bins.

Error Handling and Data Validation

Each worker, histogram writer, and reader sets a shared error flag on failure. After all output is printed, the tool throws a RuntimeException if the flag is set, which exits with a nonzero status. Validation failures for GFF, BED, and VCF set the same flag. Detected errors are reported in the output so the user can see which checks failed; the output may be incomplete when the tool exits nonzero.

Support

For questions and support:

• Email: bbushnell@lbl.gov
• Documentation: bbmap.org