Pileup2

Basic Usage

pileup2.sh in=<file,file,file> out=<file>
pileup2.sh *.sam out=<file>

Processes multiple SAM/BAM files simultaneously using multiple threads to calculate coverage statistics.

Parameters

Parameters are organized by function. Pileup2 supports all parameters from regular pileup.sh plus multi-threading specific controls.

Examples

pileup2.sh in=file1.bam,file2.bam,file3.bam out=coverage_stats.txt

pileup2.sh *.sam ref=reference.fasta out=stats.txt basecov=per_base.txt

pileup2.sh in=sample1.bam,sample2.bam,sample3.bam \
    out=stats.txt streams=8 atomic=true prealloc=true \
    32bit=true -Xmx64g

pileup2.sh in=rnaseq_*.bam stranded=true \
    out=stats_#.txt basecov=coverage_#.txt hist=histogram_#.txt

pileup2.sh in=paired_*.bam physical=true minq=20 \
    qtrim=rl trimq=10 out=physical_coverage.txt

pileup2.sh in=samples_*.bam ref=genome.fasta \
    out=stats.txt basecov=per_base.txt bincov=binned.txt \
    hist=histogram.txt rpkm=expression.txt \
    binsize=500 stdev=true countgc=true

Algorithm Details

Multi-threaded Architecture

Pileup2 implements CoveragePileupMT.LoadThread workers with ArrayBlockingQueue for concurrent processing of multiple input files:

• Stream-based Processing: Creates multiple LoadThread instances, each processing different input files simultaneously
• Producer-Consumer Pattern: Uses ArrayBlockingQueue for thread-safe file distribution among worker threads
• Thread Pool Management: Tools.mid() determines thread count based on streams parameter, available processors, and input file count
• Memory-conscious Design: Accumulates results from worker threads using synchronized accumulation to minimize memory overhead

Coverage Data Structures

Supports multiple data structures for different use cases:

• CoverageArray: CoverageArray.makeArray() creates numeric arrays storing exact coverage depths, supporting 16-bit or 32-bit precision via caType parameter
• BitSet: Java BitSet instances for memory-efficient boolean coverage tracking when only presence/absence is needed
• Atomic Arrays: Lock-free data structures using CoverageArray.getType(atomic, bits32) for high-concurrency scenarios
• Dual Strategy: Automatically selects between USE_COVERAGE_ARRAYS and USE_BITSETS based on output requirements (histogram, basecov, bincov)

Coverage Calculation Modes

Implements multiple coverage calculation strategies via processSamLine() method:

• Standard Coverage: Counts all aligned bases including insertions and deletions using addCoverage() method
• Physical Coverage: Tracks entire insert size for paired reads using pairTable HashMap, including unsequenced middle portion
• Strand-specific: Maintains separate coverage tracks using scaf.obj0 (plus strand) and scaf.obj1 (minus strand)
• Position-specific: START_ONLY and STOP_ONLY flags for tracking coverage only at read start or stop positions
• Deletion-aware: INCLUDE_DELETIONS and INCLUDE_SOFT_CLIP flags for configurable handling of deletions and soft-clipped regions

Quality Control Integration

Built-in quality control using specific classes and methods:

• Quality Trimming: TrimRead.testOptimal() and TrimRead.trimReadWithMatch() for dynamic quality-based trimming
• Mapping Quality: Configurable minMapq thresholds with sl.mapq>=minMapq filtering
• Duplicate Handling: sl.duplicate() checks with INCLUDE_DUPLICATES flag for optional PCR duplicate inclusion/exclusion
• Border Trimming: Automatic trimming near scaffold boundaries using border parameter and skipTrimRange calculations

Memory Management

Memory optimization using specific allocation strategies:

• Preallocation: Optional preallocation via prealloc parameter creates CoverageArray and BitSet instances during scaffold header processing
• Dynamic Allocation: On-demand creation using makeCA() method minimizes memory usage for sparse coverage
• Lock Optimization: Choice between synchronized blocks and atomic operations via atomic parameter based on contention patterns
• Garbage Collection Friendly: clear() method nullifies references and Tools.max() minimizes object churn during processing

Output Generation

File I/O system using specific writer classes:

• Multi-threaded Statistics: StandardDeviator.calculateStuff() with Shared.threads() for parallel calculation of coverage statistics
• Streaming Writers: ByteStreamWriter and TextStreamWriter classes with start() and poisonAndWait() lifecycle methods
• Format Flexibility: writeCoveragePerBase(), writeCoveragePerBaseBinned(), and writeRPKM() methods support multiple output formats
• Strand Separation: replaceFirst("#", "1") and replaceFirst("#", "2") for automatic filename manipulation in strand-specific outputs

Performance Characteristics

Implementation specifics for genomics workflows:

• Scalability: ThreadWaiter.startThreads() provides linear scaling with available CPU cores and input file count
• Memory Efficiency: bits32 parameter balances memory usage (16-bit vs 32-bit arrays) with maximum coverage depth (Character.MAX_VALUE vs 1000000)
• I/O Optimization: SamLineStreamer with streamerThreads parameter supports compressed formats and streaming for large datasets
• Cache Friendly: CoverageArray data structures designed for sequential access patterns during coverage accumulation

Support

For questions and support:

• Email: bbushnell@lbl.gov
• Documentation: bbmap.org