DDLWriter

Script: ddlwriter.sh Package: ddl Class: DDLWriter.java

Builds DynamicDemiLog (DDL) sketches from FASTA/FASTQ files and writes them as A48-encoded TSV. DDL sketches are compact probabilistic data structures that capture the k-mer composition of a genome, computing GC content during hashing. The resulting database files are used by QuickClade for fast taxonomic classification of unknown sequences. Supports three output modes: one sketch per file (default, multithreaded), one sketch per sequence/contig, or merged sketches per taxonomy ID.

Basic Usage

ddlwriter.sh in=<input file(s)> out=<output.tsv.gz>

Input may be one or more FASTA or FASTQ files, compressed or uncompressed, specified as a comma-delimited list. The output is a gzip-compressed TSV file containing one DDL sketch record per entry. This output serves as the reference database for QuickClade taxonomic classification.

Processing Modes

DDLWriter supports three modes that determine how input sequences are grouped into sketches:

perfile (default): Produces one DDL sketch per input file. All sequences in a file are merged into a single sketch. Supports multithreading when processing many files simultaneously.
persequence: Produces one DDL sketch per individual sequence or contig. Useful when a FASTA file contains multiple distinct organisms or when per-contig resolution is needed.
pertid: Merges all sequences sharing the same taxonomy ID into a single sketch. Requires taxonomy IDs in sequence headers or filenames. Useful for building species-level databases from multi-sequence assemblies.

Taxonomy ID Parsing

When parsetaxid=t (default), DDLWriter extracts NCBI taxonomy IDs from filenames and sequence headers. Recognized formats are tid|NNNN| or tid_NNNN_ where NNNN is the numeric taxon ID. In anonymous mode (parsetaxid=f), no taxonomy IDs are assigned and classification will use sketch similarity only without taxon labeling.

Multithreading

In perfile mode, DDLWriter processes multiple input files in parallel using all available CPU cores. Each thread independently processes one file at a time. Output records are sorted by input order for deterministic results. Thread count peaks at approximately 32 cores when processing bgzipped RefSeq-scale input due to I/O bottlenecks at higher concurrency.

Parameters

Input/Output parameters

in=<file>: Input file(s), comma-delimited. Accepts FASTA or FASTQ, compressed or uncompressed. Multiple files may also be listed as positional arguments.
out=<file>: Output DDL TSV file. Compression is inferred from the file extension; .gz is recommended (e.g., out=ddls.tsv.gz).
overwrite=f: Overwrite existing output file. By default, DDLWriter will not overwrite an existing output file.

Sketch parameters

k=31: K-mer length for hashing genome sequences into the DDL sketch. Longer k-mers provide higher specificity but reduce sensitivity for divergent sequences. Must match the k-mer length used at classification time in QuickClade.
buckets=2048: Number of DDL buckets. More buckets increase sketch resolution and cardinality estimation accuracy at the cost of larger output files and more memory per sketch. Must be a power of two.
seed=12345: Hash seed for the DDL sketch. All databases compared against each other (e.g., reference and query) must use the same seed to produce compatible sketches.
exponent=5: Exponent bit width within each 16-bit bucket value; the mantissa gets the remaining 16-exponent bits. Written to the output header and restored automatically on load. 5 suits genomes (an extra mantissa bit, and the cardinalities it gives up are unreachable); use 6 for read files.

Processing mode parameters

mode=perfile

Controls how sequences are grouped into DDL sketches. Options:

perfile: One DDL per input file (default). All sequences in a file are merged. Multithreaded when multiple files are provided.
persequence (alias: percontig): One DDL per individual FASTA/FASTQ sequence record.
pertid (alias: pertaxid): Merge all sequences sharing the same taxonomy ID into one sketch. Requires taxonomy IDs in headers or filenames.

Each value also works as a bare flag, e.g. 'pertid'.

perfile

Shorthand flag equivalent to mode=perfile.

persequence

Shorthand flag equivalent to mode=persequence. Also accepted as percontig.

pertid

Shorthand flag equivalent to mode=pertid. Also accepted as pertaxid.

Taxonomy parameters

parsetaxid=t: (parsetid) Extract taxonomy IDs from filenames or sequence headers. Recognizes the patterns tid|NNNN| and tid_NNNN_ where NNNN is a numeric NCBI taxon ID. The filename is checked first; if no ID is found there, the first sequence header in the file is checked. Set to false for anonymous mode (no taxon IDs assigned).
lineage=t: Attach a taxonomic lineage string to each record, using the tax tree. Costs a one-time tree load here, but saves every downstream reader from loading the tree at all.
usetree=t: Canonicalize each record's taxID to the tree's current node, resolving merged/synonym NCBI taxIDs to their current ID (the same remap CladeLoader applies). Keeps sketch and spectra databases agreeing on taxID so per-taxID records pair correctly. Requires the tree (shares the lineage= load); a no-op without it.
blacklist=<file>: Ignore kmers present in this file while sketching. The filename is recorded in the output header.

Advanced parameters

kmers=f: Also store the source kmers, one per bucket, in the output. Adds a data line per record; readers restore them.
threads=auto: Number of threads for perfile mode. Defaults to all available CPU cores. Effective parallelism peaks around 32 cores for bgzipped RefSeq-scale input due to decompression throughput limits.
verbose=f: Print additional progress information.

Java parameters

-Xmx: Set Java's memory allocation. The shell script defaults to 4g (-Xmx4g). For large databases with many input files, increase as needed (e.g., -Xmx16g). Memory scales with the number of simultaneously loaded sketches in pertid mode.
-eoom: Exit the process if an out-of-memory exception occurs. Requires Java 8u92 or later.
-da: Disable assertions. Can slightly increase performance in production use.

Examples

Build a reference database from multiple genome files

# One sketch per genome file (default perfile mode)
ddlwriter.sh in=genome1.fa,genome2.fa,genome3.fa out=reference.tsv.gz

# Equivalent using a file glob pattern via shell expansion
ddlwriter.sh in=$(ls genomes/*.fa | tr '\n' ',') out=reference.tsv.gz k=31 buckets=2048

The most common use case: each FASTA file represents one organism and produces one DDL record in the output. The output file is then passed to QuickClade as the reference database.

Build a per-sequence database from a multi-sequence FASTA

# One sketch per contig or sequence record
ddlwriter.sh in=all_genomes.fa out=percontig.tsv.gz mode=persequence parsetaxid=t

# Anonymous mode (no taxonomy IDs required)
ddlwriter.sh in=contigs.fa out=contigs_sketches.tsv.gz persequence parsetaxid=f

Use persequence mode when a single input file contains sequences from multiple organisms distinguished by their headers, or when building a database for contig-level classification.

Build a taxonomy-merged database from RefSeq

# Merge all sequences with the same taxonomy ID into one sketch
# Headers must contain tid|NNNN| or tid_NNNN_ patterns
ddlwriter.sh in=refseq_bacteria.fa.gz out=species_db.tsv.gz \
    mode=pertid parsetaxid=t lineage=t threads=32

# With a blacklist to remove uninformative k-mers
ddlwriter.sh in=refseq_bacteria.fa.gz out=species_db.tsv.gz \
    mode=pertid blacklist=universal_kmers.bl lineage=t

pertid mode is ideal for RefSeq-scale databases where many sequences per species exist. All sequences assigned to the same NCBI taxon ID are merged into a single representative sketch. The lineage=t flag annotates each record with its full taxonomic path (kingdom through species).

Output Format

DDLWriter produces a gzip-compressed TSV file. Each line represents one DDL sketch record with these fields:

id: Numeric identifier (file index or sequence count)
taxID: NCBI taxonomy ID, or -1 if not assigned
name: Base filename (without path or extensions) or sequence header
filename: Original input filename
bases: Total base pairs processed
contigs: Number of sequence records processed
cardinality: Estimated number of distinct k-mers
gc: GC content fraction (0.0 to 1.0)
lineage: Full taxonomic lineage string (if lineage=t and TaxTree available)
sketch: A48-encoded DDL bucket values

The file header encodes the k-mer length, seed, and blacklist used to build the database. QuickClade reads these parameters automatically to ensure compatible comparisons.

Related Tools

QuickClade — Classify unknown sequences against a DDL database produced by DDLWriter.
DDLBlacklist — Generate a blacklist of highly conserved k-mers to remove from sketches, improving classification specificity.
DDLCompare — Compare two DDL databases to measure pairwise similarity between sketches.
DDLMerger — Merge multiple DDL database files into a single consolidated database.

Support

For questions and support:

Email: bbushnell@lbl.gov
Documentation: bbmap.org