DDLMerger

Script: ddlmerger.sh Package: ddl Class: DDLMerger.java

Loads multiple DDL (Dynamic Demi-Log) database files, optionally condenses them to fewer buckets, merges records sharing a TID (taxonomy ID) — such as combining mitochondrial, plastid, or plasmid DDLs with their host genome records — then sorts, renumbers, and writes a single combined DDL file. DDL files are the compact sketch databases used by QuickClade for rapid taxonomic classification of sequencing reads.

Basic Usage

ddlmerger.sh in=a.ddl.gz,b.ddl.gz out=combined.ddl.gz

or: ddlmerger.sh in=all.ddl.gz out=filtered.ddl.gz tidsout=keep.txt \
      minsize=refseq_bacteria.fa.gz,100000+refseq_invertebrate.fa.gz,1000000

Input files are comma-delimited and may also be provided as bare positional arguments. Input and output files are typically gzip-compressed TSV files with a .ddl.gz extension.

Source Category Detection

DDLMerger automatically detects the biological category of each input file from its filename. This determines whether records from that file are eligible to merge with host genome records sharing the same taxonomy ID:

Merging Behavior

When merging is enabled (the default), records with the same taxonomy ID (TID) are combined: their DDL sketches are added together, their base counts are summed, their contig counts are summed, and their GC content is recomputed as a weighted average. GC content uses the base counts as weights, so larger genomes exert proportionally more influence on the merged GC value. After merging, all records are sorted by TID, then name, then cardinality, and renumbered sequentially before writing.

Bucket Condensing

If input DDL files were built with different (larger) bucket counts than your target, use buckets= to fold them down. Condensing takes the maximum value per destination bucket, which is conservative and preserves cardinality accuracy. Set buckets=-1 (the default) to disable condensing and keep original bucket counts.

Parameters

Input/Output parameters

in=<file,file,...>
Input DDL file(s), comma-delimited. Can also be provided as bare positional arguments on the command line. Files are typically gzip-compressed (.ddl.gz or .tsv.gz).
out=<file>
Output combined DDL file. Required. Typically gzip-compressed (.ddl.gz).
overwrite=f
Set to t to allow overwriting an existing output file. Default is false (will fail if output already exists).

Processing parameters

k=
K-mer length. Default: adopt from the input files' #k headers (all inputs must share one k, else it is a fatal error). Setting k= forces a value and validates the inputs against it. The chosen k is written to the output #k header.
buckets=-1
Target bucket count for condensing. Input DDL records with more buckets than this value will be condensed by folding (taking the maximum value per destination bucket). Set to -1 to disable condensing and preserve original bucket counts.
merge=t
Master merge switch. When true, enables merging for all categories (main genome, mitochondrial, plastid, and plasmid). Setting merge=f disables all merging; each input record will appear separately in the output. Equivalent to setting mergemito, mergeplastid, mergeplasmid, and the main-genome merge flag all at once.
mergemito=t
Merge mitochondrial records (filenames containing "mito") into host genome records sharing the same TID. Default true.
mergeplastid=t
Merge plastid records (filenames containing "plastid") into host genome records sharing the same TID. Default true.
mergeplasmid=t
Merge plasmid records (filenames containing "plasmid") into host genome records sharing the same TID. Default true.
base=<file>
Add-novel mode: keep this base DB WHOLE (every record, including the duplicate-TID mito/chloroplast/plasmid records that share a host TID), then from in= add only records whose TID is ABSENT from the base. Output = base + novel additions.
exponent=5
Exponent bit width within each 16-bit bucket value; the mantissa gets the remaining 16-exponent bits. An input file's #exponent header overrides this, and inputs that disagree with each other are a fatal error.
blacklist=<file>
Kmer blacklist; condensing prefers non-blacklisted kmers, baking the blacklist into the output.
verbose=f
Print per-file load counts and categories.
sortbytaxa=f
Order output records by taxonomy (family, then genus, then species; the same ordering as bbsort.sh taxa) instead of by cardinality.

Java Parameters

-Xmx4g
Default maximum heap size. Increase if processing very large numbers of DDL files or files with many records.
-Xms400m
Default initial heap size.
-eoom
Exit if an out-of-memory exception occurs. Requires Java 8u92+.
-da
Disable assertions.

Examples

Merge Two DDL Databases

# Combine bacteria and archaea DDL databases into one
ddlmerger.sh in=bacteria.ddl.gz,archaea.ddl.gz out=combined.ddl.gz

Records with matching TIDs from both files will be merged. Organisms appearing in both files will have their DDL sketches combined, producing a single merged record per TID.

Merge Host Genomes with Organelle Databases

# Merge main genomes with separate mitochondrial and plastid databases
ddlmerger.sh \
    in=vertebrates.ddl.gz,vertebrates_mito.ddl.gz,plants.ddl.gz,plants_plastid.ddl.gz \
    out=vertebrates_plants_combined.ddl.gz

The filenames vertebrates_mito.ddl.gz and plants_plastid.ddl.gz trigger automatic mitochondrial/plastid detection. Each organelle record will be merged into the matching host genome record based on shared TID, producing a single combined record per organism.

Condense to Fewer Buckets

# Reduce a high-resolution DDL database to 2048 buckets for faster classification
ddlmerger.sh in=big.ddl.gz out=small.ddl.gz buckets=2048 merge=f

Use merge=f when you only want to condense bucket counts without merging records by TID. Condensing uses max-per-bucket folding, which preserves cardinality accuracy while reducing file size and classification time.

Merge Multiple Databases Without Organelle Merging

# Combine databases but keep mitochondrial records separate
ddlmerger.sh in=host.ddl.gz,mito.ddl.gz out=combined.ddl.gz \
    mergemito=f

Setting mergemito=f keeps mitochondrial records as separate entries in the output, even if they share a TID with a main genome record. Useful when you want to classify mitochondrial and nuclear reads separately downstream.

Verbose Mode for Debugging Category Detection

# Check how files are being categorized
ddlmerger.sh in=a.ddl.gz,b_mito.ddl.gz,c_plastid.ddl.gz out=merged.ddl.gz \
    verbose=t

With verbose=t, DDLMerger prints the detected category (main, mitochondrion, plastid, plasmid) and record count for each input file to stderr, making it easy to confirm that filename-based category detection is working as expected.

Algorithm Details

Processing Pipeline

DDLMerger processes inputs in four stages:

  1. Load: All records from all input files are loaded into memory. Each record is tagged with its source category (main/mito/plastid/plasmid) based on filename detection. If buckets is set and a record has more buckets than the target, it is condensed immediately during loading.
  2. Merge: Records eligible for merging (based on their category flags) are grouped by TID. When multiple records share a TID, their DDL sketches are added together and their base/contig counts are accumulated. GC content is recomputed as a weighted average using base counts. Records whose category has merging disabled remain separate.
  3. Sort and renumber: All output records are sorted by TID, then by name, then by cardinality. After sorting, sequential IDs are reassigned starting from 0.
  4. Write: The combined, sorted, renumbered records are written to the output file in DDL format, with an optional blacklist header embedded if blacklist= was specified.

DDL Sketch Addition

When two DDL records are merged, their underlying DynamicDemiLog sketches are combined by adding bucket values together. This is a lossless operation: the combined sketch accurately represents the union of the k-mer sets from both records. The resulting cardinality estimate reflects the total unique k-mer content of the merged genome, organelle, and plasmid sequences.

Bucket Condensing

Condensing reduces a DDL sketch from a high bucket count to a lower one by folding: each destination bucket receives the maximum value from the corresponding source buckets. This operation is lossy (some cardinality precision is lost), but the max-per-bucket approach is conservative and tends to slightly overestimate rather than underestimate cardinality. Condensing is applied before merging.

Related Tools

Support

For questions and support: