DDLWriter

Script: ddlwriter.sh Package: ddl Class: DDLWriter.java

Builds DynamicDemiLog (DDL) sketches from FASTA/FASTQ files and writes them as A48-encoded TSV. DDL sketches are compact probabilistic data structures that capture the k-mer composition of a genome, computing GC content during hashing. The resulting database files are used by QuickClade for fast taxonomic classification of unknown sequences. Supports three output modes: one sketch per file (default, multithreaded), one sketch per sequence/contig, or merged sketches per taxonomy ID.

Basic Usage

ddlwriter.sh in=<input file(s)> out=<output.tsv.gz>

Input may be one or more FASTA or FASTQ files, compressed or uncompressed, specified as a comma-delimited list. The output is a gzip-compressed TSV file containing one DDL sketch record per entry. This output serves as the reference database for QuickClade taxonomic classification.

Processing Modes

DDLWriter supports three modes that determine how input sequences are grouped into sketches:

Taxonomy ID Parsing

When parsetaxid=t (default), DDLWriter extracts NCBI taxonomy IDs from filenames and sequence headers. Recognized formats are tid|NNNN| or tid_NNNN_ where NNNN is the numeric taxon ID. In anonymous mode (parsetaxid=f), no taxonomy IDs are assigned and classification will use sketch similarity only without taxon labeling.

Multithreading

In perfile mode, DDLWriter processes multiple input files in parallel using all available CPU cores. Each thread independently processes one file at a time. Output records are sorted by input order for deterministic results. Thread count peaks at approximately 32 cores when processing bgzipped RefSeq-scale input due to I/O bottlenecks at higher concurrency.

Parameters

Input/Output parameters

in=<file>
Input file(s), comma-delimited. Accepts FASTA or FASTQ, compressed or uncompressed. Multiple files may also be listed as positional arguments.
out=<file>
Output DDL TSV file. Compression is inferred from the file extension; .gz is recommended (e.g., out=ddls.tsv.gz).
overwrite=f
Overwrite existing output file. By default, DDLWriter will not overwrite an existing output file.

Sketch parameters

k=31
K-mer length for hashing genome sequences into the DDL sketch. Longer k-mers provide higher specificity but reduce sensitivity for divergent sequences. Must match the k-mer length used at classification time in QuickClade.
buckets=2048
Number of DDL buckets. More buckets increase sketch resolution and cardinality estimation accuracy at the cost of larger output files and more memory per sketch. Must be a power of two.
seed=12345
Hash seed for the DDL sketch. All databases compared against each other (e.g., reference and query) must use the same seed to produce compatible sketches.
exponent=<int>
(ebits) Sets the DynamicDemiLog exponent controlling the precision of the estimator's floating-point representation. Advanced parameter; the default is appropriate for most uses.

Processing mode parameters

mode=perfile
Controls how sequences are grouped into DDL sketches. Options:
  • perfile: One DDL per input file (default). All sequences in a file are merged. Multithreaded when multiple files are provided.
  • persequence (alias: percontig): One DDL per individual FASTA/FASTQ sequence record.
  • pertid (alias: pertaxid): Merge all sequences sharing the same taxonomy ID into one sketch. Requires taxonomy IDs in headers or filenames.
perfile
Shorthand flag equivalent to mode=perfile.
persequence
Shorthand flag equivalent to mode=persequence. Also accepted as percontig.
pertid
Shorthand flag equivalent to mode=pertid. Also accepted as pertaxid.

Taxonomy parameters

parsetaxid=t
(parsetid) Extract taxonomy IDs from filenames or sequence headers. Recognizes the patterns tid|NNNN| and tid_NNNN_ where NNNN is a numeric NCBI taxon ID. The filename is checked first; if no ID is found there, the first sequence header in the file is checked. Set to false for anonymous mode (no taxon IDs assigned).
lineage=t
(writelineage) Attach full taxonomic lineage strings to DDL records that have a valid taxonomy ID. Requires a TaxTree to be loaded. If the TaxTree cannot be loaded, a warning is printed and lineage is skipped. Lineage data enables more informative output in downstream QuickClade classification.
blacklist=<file>
A file of k-mers (or a DDL blacklist file) to exclude from all sketches. K-mers in the blacklist are ignored during hashing. Used to remove highly conserved or uninformative k-mers (e.g., universal repeats) that reduce classification specificity. See DDLBlacklist for generating blacklist files.

Advanced parameters

kmers=f
(storekmers) Store the actual k-mer sequences in the DDL sketch in addition to the hashed bucket values. Increases memory and file size substantially. Rarely needed for standard classification workflows.
threads=auto
Number of threads for perfile mode. Defaults to all available CPU cores. Effective parallelism peaks around 32 cores for bgzipped RefSeq-scale input due to decompression throughput limits.
verbose=f
Print the filename of each input file as it is processed. Useful for monitoring progress when processing many files.

Java parameters

-Xmx
Set Java's memory allocation. The shell script defaults to 4g (-Xmx4g). For large databases with many input files, increase as needed (e.g., -Xmx16g). Memory scales with the number of simultaneously loaded sketches in pertid mode.
-eoom
Exit the process if an out-of-memory exception occurs. Requires Java 8u92 or later.
-da
Disable assertions. Can slightly increase performance in production use.

Examples

Build a reference database from multiple genome files

# One sketch per genome file (default perfile mode)
ddlwriter.sh in=genome1.fa,genome2.fa,genome3.fa out=reference.tsv.gz

# Equivalent using a file glob pattern via shell expansion
ddlwriter.sh in=$(ls genomes/*.fa | tr '\n' ',') out=reference.tsv.gz k=31 buckets=2048

The most common use case: each FASTA file represents one organism and produces one DDL record in the output. The output file is then passed to QuickClade as the reference database.

Build a per-sequence database from a multi-sequence FASTA

# One sketch per contig or sequence record
ddlwriter.sh in=all_genomes.fa out=percontig.tsv.gz mode=persequence parsetaxid=t

# Anonymous mode (no taxonomy IDs required)
ddlwriter.sh in=contigs.fa out=contigs_sketches.tsv.gz persequence parsetaxid=f

Use persequence mode when a single input file contains sequences from multiple organisms distinguished by their headers, or when building a database for contig-level classification.

Build a taxonomy-merged database from RefSeq

# Merge all sequences with the same taxonomy ID into one sketch
# Headers must contain tid|NNNN| or tid_NNNN_ patterns
ddlwriter.sh in=refseq_bacteria.fa.gz out=species_db.tsv.gz \
    mode=pertid parsetaxid=t lineage=t threads=32

# With a blacklist to remove uninformative k-mers
ddlwriter.sh in=refseq_bacteria.fa.gz out=species_db.tsv.gz \
    mode=pertid blacklist=universal_kmers.bl lineage=t

pertid mode is ideal for RefSeq-scale databases where many sequences per species exist. All sequences assigned to the same NCBI taxon ID are merged into a single representative sketch. The lineage=t flag annotates each record with its full taxonomic path (kingdom through species).

Output Format

DDLWriter produces a gzip-compressed TSV file. Each line represents one DDL sketch record with these fields:

The file header encodes the k-mer length, seed, and blacklist used to build the database. QuickClade reads these parameters automatically to ensure compatible comparisons.

Related Tools

Support

For questions and support: