FilterVCF

Script: filtervcf.sh Package: var2 Class: FilterVCF.java

Filters VCF files by position, variant type, or quality attributes. Supports BED-file region restriction, indel left-alignment/normalization, genotype filtering, multi-allelic splitting, and multithreaded processing. Filtering by optional quality fields requires VCF files generated by CallVariants.

Basic Usage

filtervcf.sh in=<file> out=<file>

FilterVCF processes VCF files to remove variants that don't meet specified quality, position, type, or genotype criteria. It can operate in single-threaded or multithreaded mode for better performance on large files. Filtering by optional fields such as allele fraction and mapping quality requires VCF files produced by CallVariants.

Parameters

Parameters are organized by filtering function. The tool uses both the VarFilter system (variant-specific quality criteria) and the SamFilter system (position-specific criteria) for comprehensive variant evaluation.

I/O parameters

in=<file>: Input VCF file. Required.
out=<file>: Output VCF file. If not specified, output goes to stdout.
ref=<file>: Reference fasta (optional). Required when using normalize=t for indel left-alignment. Also used for coordinate resolution via ScafMap.
overwrite=f: (ow) Set to false to abort rather than overwrite an existing output file. Default: true.
bgzip=f: Use bgzip for gzip compression instead of standard gzip. Produces BGZF-format output compatible with tabix indexing. Default: false.
splitalleles=f: Split multi-allelic lines into one line per alternate allele. Correctly handles FORMAT field data for each allele. Default: false.
splitsubs=f: (splitsnps) Split multi-base substitutions into individual SNPs. Useful for downstream analysis requiring single-nucleotide variants. Default: false.
splitcomplex=f: Split complex variants into simpler canonical forms. Default: false.
split=f: (sass) Convenience flag: sets both splitalleles=t and splitsubs=t simultaneously. Default: false.
splitall=f: (sascsss) Convenience flag: sets splitalleles=t, splitsubs=t, and splitcomplex=t simultaneously. Default: false.
canonize=t: (canonicalize, trimtocanonical) Trim variations down to a canonical representation by removing redundant flanking bases. Default: true.
normalize=f: (leftalign, norm) Left-align indels using the reference sequence (requires ref=). Combined with splitalleles=t, this produces a clean canonical VCF equivalent to 'bcftools norm -m -both'. Default: false.
mt=f: (multithreaded) Enable multithreaded processing for faster performance on large files. Up to 8 threads are used. Single-threaded mode is the default. Default: false.
scorehist=<file>: (qualhist, qhist, shist) Write a histogram of variant quality scores to this file for distribution analysis.

Position-filtering parameters

minpos=: Ignore variants that do not overlap this minimum position. Coordinate-based filtering via the SamFilter system.
maxpos=: Ignore variants that do not overlap this maximum position. Works together with minpos= to define a coordinate window.
contigs=: Comma-delimited list of contig names to include. Names should have no spaces; use underscores instead.
invert=f: Invert position filters. When true, keeps variants outside the specified coordinate range or contig list instead of inside. Default: false.
bed=<file>: Keep only variants whose position falls inside an interval in this BED file (e.g., a high-confidence benchmark region set such as GIAB). Intervals are loaded into a BedMask at startup; the number of intervals loaded is reported to stderr. Default: none.
invertbed=f: (excludebed, bedexclude, bedinvert) Invert the BED filter: keep only variants whose position falls OUTSIDE the BED intervals. Default: false.

Type-filtering parameters

sub=t: (subs) Keep substitutions (SNPs). Default: true.
del=t: (dels) Keep deletions. Default: true.
ins=t: (inss) Keep insertions. Default: true.
indel=t: (indels) Convenience flag: sets both ins= and del= to the same value. Default: true.
junction=t: (junctions) Keep junction variants (structural variants detected as splice junctions). Default: true.

Genotype-filtering parameters

gt=: (genotype) Comma-delimited whitelist of genotype strings to keep (e.g., gt=1/1 or gt=0/1,1/1). Phased genotypes using '|' are normalized to '/' for matching. Variants lacking a sample/GT column are rejected when this filter is active. Setting gt= to nothing, "all", or "any" disables genotype filtering. Default: no filter.
homozygous=f: (hom) When true, keep only homozygous variants (all alleles identical). Haploid variants are considered homozygous. Variants with missing alleles ('.') are excluded. Default: false.
heterozygous=f: (het) When true, keep only heterozygous variants (at least two differing alleles). Mutually exclusive with homozygous=t in effect. Default: false.

Variant-quality filtering parameters

These parameters filter on statistics stored in the INFO and FORMAT fields of VCF files produced by CallVariants. They have no effect on VCF files from other callers that lack these fields. All thresholds are minimums unless noted otherwise.

minreads=0: Ignore variants seen in fewer reads. Minimum coverage (read depth) for variant support. Default: 0.
minqualitymax=0: Ignore variants with lower max base quality. Filters on the highest quality base supporting the variant. Default: 0.
minedistmax=0: Ignore variants with lower max distance from read ends. Filters variants too close to read termini, which are error-prone regions. Default: 0.
minmapqmax=0: Ignore variants with lower max mapping quality. Filters based on the best mapping quality among supporting reads. Default: 0.
minidmax=0: Ignore variants with lower max read identity. Filters variants found only in reads with poor overall identity to reference. Default: 0.
minpairingrate=0.0: Ignore variants with lower pairing rate. Requires proper-pair information from alignment. Range: 0.0–1.0. Default: 0.0.
minstrandratio=0.0: Ignore variants with lower plus/minus strand ratio. Filters variants with strong strand bias, which often indicates sequencing artifacts. Default: 0.0.
minquality=0.0: Ignore variants with lower average base quality. Uses the mean quality of all bases supporting the variant. Default: 0.0.
minedist=0.0: Ignore variants with lower average distance from read ends. Mean distance from termini for variant-supporting bases. Default: 0.0.
minavgmapq=0.0: Ignore variants with lower average mapping quality. Mean mapping quality across all supporting reads. Default: 0.0.
minallelefraction=0.0: Ignore variants with lower allele fraction. Should be adjusted for ploidy: e.g., ~0.1 for diploid, ~0.05 for tetraploid. Default: 0.0.
minid=0: Ignore variants with lower average read identity. Mean identity score across all supporting reads. Default: 0.
minscore=0.0: Ignore variants with lower Phred-scaled score (the VCF QUAL column). Simple quality threshold applied directly to the QUAL field without statistical recomputation. Default: 0.0.
clearfilters: Reset all variant filters to zero and all position filters to their defaults. Clears both VarFilter and SamFilter criteria.

There are additionally maximum filters for score, quality, mapping quality, allele fraction, and identity. These are available through VarFilter and may be set directly (e.g., maxallelefraction=0.9).

Neural network parameters

FilterVCF can use a trained neural network to score variants, providing a learned quality metric that integrates many features simultaneously.

net=<file>: (netfile) Load a neural network model file for variant scoring. When specified, enables neural network filtering. Default: none.
netcutoff=auto: Score threshold for neural network filtering. Variants scoring below this cutoff are removed. Set to "auto" to use the cutoff stored in the network file, or provide a numeric value to override. Default: auto.
usenet=f: (useann, usenn, nn) Enable neural network scoring without specifying a net= file (uses a default network if available). Default: false.
netmode=: Set the feature extraction mode for the neural network. Also enables the network when specified. Default: not set.

Java Parameters

-Xmx: Set Java's memory usage, overriding autodetection. -Xmx20g specifies 20 GB; -Xmx200m specifies 200 MB. The default is 4 GB. Maximum is typically 85% of physical memory.
-eoom: Exit if an out-of-memory exception occurs. Requires Java 8u92+.
-da: Disable assertions. Can provide minor performance improvements.

Examples

Basic Quality Filtering

filtervcf.sh in=variants.vcf out=filtered.vcf minscore=20.0 minallelefraction=0.1

Filter variants requiring a minimum QUAL score of 20 and allele fraction of at least 10%. Suitable for diploid organisms with CallVariants output.

BED Region Restriction (High-Confidence Regions)

filtervcf.sh in=variants.vcf out=hc_variants.vcf bed=HG001_GRCh38_GIAB_highconf.bed

Keep only variants falling inside GIAB high-confidence benchmark regions. Useful for benchmarking variant callers against truth sets.

BED Region Exclusion

filtervcf.sh in=variants.vcf out=outside_repeats.vcf bed=repeatmasker.bed invertbed=t

Remove variants that fall inside repeat-masked regions. The invertbed=t flag keeps only variants outside the BED intervals.

Normalization and Splitting

filtervcf.sh in=variants.vcf out=normalized.vcf ref=genome.fa normalize=t splitalleles=t

Left-align indels against the reference and split multi-allelic sites into biallelic records. Equivalent to 'bcftools norm -m -both'. Requires ref= for normalization.

Position-based Filtering

filtervcf.sh in=variants.vcf out=region.vcf minpos=1000000 maxpos=2000000 contigs=chr1,chr2

Extract variants from chromosomes 1 and 2 within the coordinate range 1 Mb to 2 Mb.

Type-specific Filtering

filtervcf.sh in=variants.vcf out=snps_only.vcf del=f ins=f sub=t

Keep only substitutions (SNPs), filtering out all insertions and deletions.

Genotype Filtering

filtervcf.sh in=variants.vcf out=homozygous.vcf homozygous=t

Keep only homozygous variants (all alleles identical). Haploid variants are treated as homozygous.

Specific Genotype Whitelist

filtervcf.sh in=variants.vcf out=het_only.vcf gt=0/1

Keep only heterozygous alt variants with genotype exactly 0/1. Phased genotypes using '|' are normalized to '/' for matching.

Comprehensive Quality Control

filtervcf.sh in=variants.vcf out=highqual.vcf minreads=5 minquality=25.0 minstrandratio=0.2 minedist=10.0 minallelefraction=0.15 bed=highconf.bed

Stringent filtering requiring: minimum 5 supporting reads, average base quality ≥25, strand ratio ≥0.2, distance from read ends ≥10 bp, allele fraction ≥15%, and restriction to high-confidence regions.

Multithreaded Processing

filtervcf.sh in=large_cohort.vcf out=filtered.vcf mt=t minscore=30 minallelefraction=0.05

Enable multithreaded mode for faster processing of large VCF files. Up to 8 threads are used. The output order is preserved.

Algorithm Details

Filtering Architecture

FilterVCF implements a layered filter pipeline. Variants must pass all active filters in order to be written to output:

Quality score (QUAL): Direct threshold on the VCF QUAL column (minscore=, minqual=).
Type filtering: Variant type inclusion/exclusion (sub=, del=, ins=, junction=).
Position filtering (SamFilter): Coordinate range, contig whitelist (minpos=, maxpos=, contigs=, invert=).
Region filtering (BedMask): Interval-based inclusion or exclusion using a BED file (bed=, invertbed=).
Genotype filtering: GT field whitelist and zygosity constraints (gt=, homozygous=, heterozygous=).
Statistical filtering (VarFilter): Coverage depth, base quality, strand bias, mapping quality, and allele fraction thresholds. Requires CallVariants-format INFO fields.

BedMask Region Filtering

When bed= is specified, a BedMask is loaded from the BED file at startup. The number of intervals and scaffolds loaded is reported to stderr. Each variant's chromosome and position are tested against the loaded intervals. By default, only variants inside the intervals are kept. Setting invertbed=t keeps only variants outside the intervals (e.g., to exclude repeat regions).

Normalization (Left-Alignment)

When normalize=t is set, indels are left-aligned against the reference sequence after filtering and before output. This produces a canonical representation where all equivalent indel representations are collapsed to the leftmost position. Requires ref= to be specified. Combined with splitalleles=t, this matches the behavior of 'bcftools norm -m -both'.

Variant Splitting

Three decomposition modes are supported, applied after all filters pass:

Allele splitting (splitalleles=t): Multi-allelic records (e.g., A>G,T) are split into separate biallelic records (A>G and A>T), with FORMAT fields correctly handled per allele.
Substitution splitting (splitsubs=t): Multi-base substitutions are decomposed into constituent SNPs where possible.
Complex splitting (splitcomplex=t): Complex variants are further decomposed into simpler canonical forms.

Threading

In multithreaded mode (mt=t), up to 8 worker threads process variant batches in parallel. The input ByteFile read is serialized under a lock so that decompression and parsing are parallel while the shared buffer is accessed safely. Each thread maintains independent statistics that are aggregated at completion. Output ordering is preserved via job-ID sequencing.

Related Tools

callvariants.sh — Variant caller that produces VCF files with the INFO fields required by FilterVCF's statistical filters.
comparevcf.sh — Compares two VCF files (e.g., a call set vs. a truth set), reporting precision, recall, and F-measure. Also supports normalization features shared with FilterVCF.
gradevcf.sh — Grades a VCF file against a truth set to assess variant calling accuracy.
invertvcf.sh — Inverts a VCF file by computing the complement: all positions not called as variants.

Support

For questions and support:

Email: bbushnell@lbl.gov
Documentation: bbmap.org