BalanceVectors

Basic Usage

balancevectors.sh in=<vectors.tsv> outtrain=<file> outval=<file>

The input must be a tab-delimited vector file in the format produced by VcfToTrainingVectors, where the last column is a 0/1 label (1 = true variant, 0 = false positive). The tool writes two output files: a training split and a held-out validation split.

Sampling Strategy

Balancing proceeds in five steps, all deterministic given seed:

1. Load and partition: All positives (label > 0.5) are kept in full. Negatives are separated for sampling.
2. Target count: The number of negatives to select is chosen so that positives represent posfraction of the final output (default 30%).
3. Enriched negatives: A per-category quota is drawn across existing axes (variant type × depth, score, event length; plus depth alone). If newcats=t, additional artifact axes (type × homopolymer run, allele fraction bins, strand ratio bins, mapping quality bins) are sampled at newweight times the base quota.
4. Representative negatives: The remainder of the target is filled from a random sample of negatives not already selected by enrichment.
5. Shuffle and split: Positives and chosen negatives are combined, globally shuffled, and split: the last valfraction of rows go to the validation file and the rest to training.

Memory Management

BalanceVectors loads all input vectors into memory at once. Budget approximately 1.5× the input file size in heap. The default JVM allocation is 4 GB; override with -Xmx for large inputs (tens of millions of vectors).

Parameters

Examples

# Balance a vector file and split into 90% train / 10% validation
balancevectors.sh in=variants.tsv outtrain=train.tsv outval=val.tsv

# 40% positives, 15% validation hold-out, reproducible with a different seed
balancevectors.sh in=variants.tsv outtrain=train.tsv outval=val.tsv \
    posfraction=0.4 valfraction=0.15 seed=1234

# Disable stratified enrichment entirely for a flat random negative sample
balancevectors.sh in=variants.tsv outtrain=train.tsv outval=val.tsv enrich=f

# Enrichment on but artifact axes off (use only type/depth/score/length axes)
balancevectors.sh in=variants.tsv outtrain=train.tsv outval=val.tsv newcats=f

Algorithm Details

Category Axes

BalanceVectors builds category keys from the VectorUMP45 33-feature layout. Each negative can appear in multiple buckets simultaneously; the enrichment pass deduplicates chosen indices so a single vector is never double-counted.

Existing axes (always used when enrich=t):

• Type × depth: {SUB, INS, DEL, OTHER} × logarithmically binned read depth
• Type × score: {SUB, INS, DEL, OTHER} × composite-score bins (0–19)
• Type × event length: {SUB, INS, DEL, OTHER} × {len1, len2-5, len6-20, len20+}
• Depth alone: logarithmically binned depth without variant type

Artifact axes (used when newcats=t, sampled at newweight × base quota):

• Type × homopolymer: Slippage-prone sites; homopolymer run length encoded as 1/(run+1) and inverted
• Allele fraction: Bins {0–10%, 10–30%, 30–60%, 60%+}; low-VAF errors concentrate below 10%
• Strand ratio: Bins {0–10%, 10–30%, 30%+}; extreme strand bias indicates artifact
• Mapping quality: Bins {0–10, 10–20, 20–40, 40+}; low-MAPQ reads are enriched for false positives

Sampling Procedure

Within each category bucket the tool applies a partial Fisher-Yates shuffle (Knuth) to draw up to quota indices uniformly without replacement. Because each negative index can appear in many buckets, a global chosen[] boolean array ensures that the total enriched set contains distinct vectors. The representative fill then draws uniformly from all unchosen negatives using another partial shuffle, pinning the final positive fraction to posfraction.

Score Column Zeroing

With noscore=t (default), column 29 (the CallVariants composite score) is replaced with the literal character 0 on output. The zeroing is done at the byte level during the write pass — no floating-point parsing — so the remainder of each line is reproduced exactly. This is the recommended setting when training a replacement scoring model: the network cannot shortcut learning by copying a preexisting signal.

Related Tools

• CallVariants — Calls SNPs and indels from aligned reads; the variant call output feeds into VcfToTrainingVectors to produce the labeled vector files that BalanceVectors consumes.
• VcfToTrainingVectors — Converts a VCF and its associated BAM/SAM into the labeled TSV vector format that BalanceVectors expects as input.

Support

For questions and support:

• Email: bbushnell@lbl.gov
• Documentation: bbmap.org