CovMaker

Basic Usage

covmaker.sh *.sam out=cov.txt
covmaker.sh in=sample1.bam,sample2.bam out=cov.txt
covmaker.sh cov.txt out=cov7.txt condense=7

CovMaker accepts three types of input files: SAM (Sequence Alignment Map), BAM (Binary Alignment Map), or existing COV (coverage) files. Input format is auto-detected by file extension and content. Output is always a COV file containing depth matrices suitable for genomic binning analysis.

Memory Management

CovMaker can use significant memory when processing many contigs and samples. Memory usage is primarily driven by: (1) number of samples, (2) number of unique contigs, and (3) proxy set size for similarity calculations. SAM files use substantially less memory than BAM files, and COV files use the least memory since data is already structured. Without -Xmx, the heap is set to 84% of available memory (on Linux, MemAvailable minus 500 MB reserved; memdetect.sh estimates lower on macOS), with a 4000 MB fallback if detection fails. This can be overridden with Java parameters.

Threading and Concurrency

CovMaker loads SAM/BAM files with multiple concurrent readers (readthreads, default 4 files at once). This parallel loading reduces total processing time but increases peak memory usage. Reduce readthreads if memory becomes a limiting factor. The condensing operation itself is single-threaded but highly optimized with SIMD calculations.

Common Use Cases

Converting Alignment Files to Coverage Matrix

The basic workflow: align reads to contigs using BBMap or similar tool, then convert the resulting SAM or BAM file to a coverage matrix for binning analysis.

# Convert SAM file to coverage format
covmaker.sh alignments.sam out=coverage.cov

# Convert multiple BAM files (higher memory usage)
covmaker.sh in=sample1.bam,sample2.bam,sample3.bam out=coverage.cov

# Process with minimum contig filtering
covmaker.sh in=alignments.sam out=coverage.cov mincontig=500

Condensing Many Samples into Fewer Logical Samples

When you have too many samples for efficient binning (typically >10-20), merge similar samples to reduce dimensionality while preserving binning quality. CovMaker uses cosine similarity on depth vectors to intelligently group similar samples.

# Condense 50 samples down to 10 using similarity merging
covmaker.sh in=all_samples.cov out=condensed.cov condense=10

# Aggressive condensing with entropy weighting
covmaker.sh in=alignments.bam out=merged.cov condense=7 entropy=t magnitude=t

# Let the data choose the target (correlation-based estimate)
covmaker.sh in=all_samples.cov out=auto.cov condense=auto

Sample Reordering for Improved Indexing

Reorder samples by depth entropy to place high-information samples at indices 0, 1, 2. This improves QuickBin's internal indexing and can significantly improve binning accuracy for the primary samples of interest.

# Reorder samples by decreasing entropy (default behavior)
covmaker.sh in=coverage.cov out=reordered.cov reorder=t

# Combine condensing and reordering
covmaker.sh in=many_samples.cov out=optimized.cov condense=15 reorder=t

Complete QuickBin Workflow

Integration with downstream QuickBin analysis shows the typical usage context.

# 1. Align reads to assembled contigs
bbmap.sh ref=contigs.fa in=reads_R1.fastq in2=reads_R2.fastq out=alignment.sam

# 2. Convert to optimized coverage matrix
covmaker.sh alignment.sam out=coverage.cov mincontig=1000 reorder=t

# 3. Run QuickBin binning (separate tool)
quickbin.sh contigs=contigs.fa cov=coverage.cov out=bins.txt

Subset Extraction from Existing Coverage File

Extract coverage information for a specific set of contigs using a reference FASTA file, creating a condensed coverage matrix.

# Extract coverage only for contigs in subset.fa
covmaker.sh ref=subset.fa in=full_coverage.cov out=subset_coverage.cov mincontig=500

Parameters

CovMaker parameters are organized by function. Input can be SAM/BAM/COV files or comma-separated lists. Parameters control input format detection, output file handling, sample merging strategy, and memory/threading behavior.

Algorithm Details

Coverage Matrix Format

CovMaker generates COV files containing a matrix where rows are contigs and columns are samples. Each cell contains the average depth (coverage) of that contig in that sample. The format is designed for efficient access by genomic binning algorithms.

Depth for each contig and sample is the sum of counted aligned bases divided by contig length. Alignments with identity below 0.96 (minid default) are not counted. For contigs of at least 150 bp (1.5 x tipLimit, default tipLimit 100), only the part of each alignment inside the central region is counted: the region excludes min(100, length/4) bases at each end, and the overlap is scaled by contig length divided by central-region length. Mapping quality below 20 (mapq default) excludes a read only from pair-graph edges, not from depth.

Sample Condensing Strategy

Condensing uses a greedy merging algorithm optimized with pre-calculated proxy matrices:

• Proxy matrix construction: Uses the top N contigs (by size) to represent each sample's coverage profile, reducing memory from O(samples × contigs) to O(samples × N)
• Similarity calculation: Cosine similarity computed on proxy matrices; log-normalized versions pre-calculated if lognorm=t
• Greedy merging: Repeatedly finds the two samples with minimum merge cost and combines them until target count reached
• Cost function: cost(i,j) = cosine term × volume term × entropy term. The cosine term is 1 - cosine similarity when cosine=t, and 1 otherwise; with negcos=t it becomes 1.0001 minus that value. The volume term is (volume_i + volume_j)^magpower, then square-rooted when rootmagnitude=t, and 1 when magnitude=f; volume_i is the sum of depths over the proxy contigs. The entropy term is (entropy_i + entropy_j)^entpower, and 1 when entropy=f; each sample's entropy is the sum of log(depth+1) over proxy contigs with nonzero depth, divided by the number of proxy contigs. The lowest-cost pair merges first.
• Full-data application: After determining merge order on proxy matrices, merges are applied to complete coverage data for all contigs

Automatic Target Selection (condense=auto)

With condense=auto, the target count is chosen before the greedy merge runs:

• Correlation: Pearson r between each pair of samples, computed on log(depth+0.5) over the largest contigs (up to compare). Rows where both samples have zero depth are excluded for that pair.
• Merge distances: Single-linkage clustering on distance 1 - r. The n-1 merge distances that join components are sorted in ascending order.
• Elbow: The cut point is the largest ratio between successive merge distances, if that ratio exceeds gapratio (default 3.0). Merges up to the cut point are accepted. If no ratio exceeds gapratio, only the alwaysmerge merges are accepted.
• Guard bands: Merges with r at or above alwaysmerge (default 0.95) are always accepted. Merges with r at or below nevermerge (default 0.70) are never accepted. The accepted count is clamped to this range.
• Result: Target = input samples minus accepted merges. If the target equals the input count, nothing is condensed.

Sample Reordering

Entropy-based reordering improves QuickBin's internal indexing by placing most informative samples at the beginning:

• Entropy calculation: For each sample, sums log(depth+1) over contigs with nonzero depth and size >= minseed (default 2500), using at most the first 100,000 contigs in working order
• Sorting: Samples sorted in descending entropy order
• Permutation: Depth values for all contigs reorganized to match new sample order
• Impact: First few samples (highest entropy) receive optimized processing in downstream binning

Memory Characteristics

Peak memory usage depends on operation:

• Loading SAM: Holds all contigs and depths in memory; scales with contig count
• Loading BAM: Same memory plus BGZF decompression overhead; typically 2-4x higher than SAM
• Condensing: Builds cost matrix O(samples^2) plus proxy matrices; dominates for many samples
• Reading COV: Minimal memory; can process COV files with limited resources

SIMD Acceleration

Vector operations for cosine similarity calculations use SIMD (Single Instruction, Multiple Data) instructions when available, accelerating the similarity matrix computation phase. SIMD is enabled automatically when the CPU reports AVX2, AVX-512, or SVE and Java is version 17 or later (javasetup.sh).

Workflow Integration

Alignment to Binning Pipeline

CovMaker serves as the bridge between alignment and genomic binning:

• Input source: SAM/BAM files from sequence alignment (BBMap, BWA, Bowtie2, etc.)
• Coverage extraction: Parses CIGAR strings and position information to calculate per-contig depths
• Output format: Optimized coverage matrix readable by QuickBin and other binning tools
• Optimization: Optional sample merging and reordering to improve downstream binning quality

Multi-Sample Metagenomics Workflows

For metagenomic assembly with read samples from multiple sources or conditions:

• Single analysis: Combine all samples in one alignment, process with CovMaker using condense if needed
• Subset extraction: Generate coverage for full assembly, then extract subsets for specific sample groups using ref parameter
• Quality control: Monitor sample reordering to identify outlier or low-quality samples (very low entropy)

Integration with QuickBin

CovMaker output is specifically designed for QuickBin genomic binning:

• Format compatibility: COV format is native input for QuickBin
• Sample ordering: Entropy-based reordering aligns with QuickBin's preference for high-variance samples at early indices
• Condensing benefit: Reduces QuickBin runtime and memory while preserving binning accuracy
• Typical workflow: Align reads → covmaker → QuickBin → binning results

Performance Characteristics

Speed

Performance depends primarily on input size and format:

• SAM parsing: ~50-200k reads/second per thread, depending on alignment complexity
• BAM decompression: ~100-300k reads/second when using multiple concurrent readers
• Condensing: O(samples^2 × proxy_size); ~10 samples/minute for typical datasets
• Scaling: Linear with number of samples; quadratic with contig count only for condense operation

Disk I/O

For large datasets with many samples:

• Input: SAM/BAM files should be on fast storage or SSD for optimal throughput
• Output: COV files are typically 10-100x smaller than input BAM files
• Parallelization: Multiple SAM files can be processed in parallel with readthreads parameter

Examples

# Simple SAM to coverage conversion
covmaker.sh reads_aligned.sam out=coverage.cov

# From multiple BAM files
covmaker.sh in=sample1.bam,sample2.bam,sample3.bam out=combined.cov

# Only include contigs >= 1 kb
covmaker.sh alignments.sam out=coverage.cov mincontig=1000

# Very stringent filtering (only > 5 kb contigs)
covmaker.sh alignments.bam out=coverage.cov mincontig=5000

# Condense 20 samples to 10 using cosine similarity
covmaker.sh alignments.bam out=condensed.cov condense=10

# Aggressive condensing with multiple weighting metrics
covmaker.sh alignments.bam out=condensed.cov condense=5 cosine=t magnitude=t entropy=t

# Reorder samples by entropy (default)
covmaker.sh coverage.cov out=reordered.cov reorder=t

# Reorder with custom entropy threshold
covmaker.sh coverage.cov out=reordered.cov reorder=t minseed=5000

# Merge samples AND reorder for maximum optimization
covmaker.sh many_samples.bam out=optimized.cov condense=12 reorder=t magnitude=t cosine=t

# With memory constraints
covmaker.sh -Xmx16g alignments.bam out=result.cov condense=8 readthreads=2 compare=50000

# Extract coverage only for contigs in bin_genomes.fa
covmaker.sh ref=bin_genomes.fa in=full_coverage.cov out=bin_coverage.cov

# With contig filtering
covmaker.sh ref=target_contigs.fa in=alignments.sam out=subset.cov mincontig=2000

# Read from COV (uses minimal memory), then reorder
covmaker.sh large_coverage.cov out=reordered.cov reorder=t

# Read from SAM with reduced threading to lower peak memory
covmaker.sh -Xmx8g alignments.sam out=coverage.cov readthreads=1 mincontig=1000

# Condense from COV input (very efficient)
covmaker.sh -Xmx4g full.cov out=condensed.cov condense=7

# Step 1: Align all reads to assembly
bbmap.sh ref=assembly.fa in=reads_R1.fastq in2=reads_R2.fastq out=alignment.sam

# Step 2: Optimize coverage matrix
covmaker.sh alignment.sam out=coverage.cov mincontig=1000 condense=15 reorder=t

# Step 3: Run genomic binning
quickbin.sh contigs=assembly.fa cov=coverage.cov out=bins.txt

# Step 4: Prepare genome-specific coverage (optional subset analysis)
covmaker.sh ref=bin1_contigs.fa in=coverage.cov out=bin1_cov.cov

Troubleshooting

Out of Memory Errors

If you encounter "OutOfMemoryException":

• Increase -Xmx parameter: covmaker.sh -Xmx32g alignments.bam out=coverage.cov
• Use SAM instead of BAM: SAM files use 50% less memory than BAM during parsing
• Increase mincontig to exclude small sequences: mincontig=5000
• Reduce readthreads to lower peak concurrent memory: readthreads=1
• If condensing, reduce maxsamples or condense to fewer samples

Slow Condensing with Many Samples

If condensing takes very long:

• Reduce compare parameter to use fewer contigs for similarity: compare=50000 instead of 100000
• Pre-condense to intermediate target first, then final target
• Read from COV format instead of SAM/BAM to skip alignment parsing

Small or Empty Coverage Values

If contigs have very low coverage:

• Verify alignment quality and read counts
• Check if sequences were actually aligned (verify SAM/BAM file size)
• Confirm reference assembly matches the aligned reads

Support

For questions and support:

• Email: bbushnell@lbl.gov
• Documentation: bbmap.org
• For QuickBin integration questions, see QuickBin documentation