DDLMerger

Script: ddlmerger.sh Package: ddl Class: DDLMerger.java

Loads multiple DDL (Dynamic Demi-Log) database files, optionally condenses them to fewer buckets, merges records sharing a TID (taxonomy ID) — such as combining mitochondrial, plastid, or plasmid DDLs with their host genome records — then sorts, renumbers, and writes a single combined DDL file. DDL files are the compact sketch databases used by QuickClade for rapid taxonomic classification of sequencing reads.

Basic Usage

ddlmerger.sh in=a.ddl.gz,b.ddl.gz out=combined.ddl.gz

Input files are comma-delimited and may also be provided as bare positional arguments. Input and output files are typically gzip-compressed TSV files with a .ddl.gz extension.

Source Category Detection

DDLMerger automatically detects the biological category of each input file from its filename. This determines whether records from that file are eligible to merge with host genome records sharing the same taxonomy ID:

Merging Behavior

When merging is enabled (the default), records with the same taxonomy ID (TID) are combined: their DDL sketches are added together, their base counts are summed, their contig counts are summed, and their GC content is recomputed as a weighted average. GC content uses the base counts as weights, so larger genomes exert proportionally more influence on the merged GC value. After merging, all records are sorted by TID, then name, then cardinality, and renumbered sequentially before writing.

Bucket Condensing

If input DDL files were built with different (larger) bucket counts than your target, use buckets= to fold them down. Condensing takes the maximum value per destination bucket, which is conservative and preserves cardinality accuracy. Set buckets=-1 (the default) to disable condensing and keep original bucket counts.

Parameters

Input/Output parameters

in=<file,file,...>
Input DDL file(s), comma-delimited. Can also be provided as bare positional arguments on the command line. Files are typically gzip-compressed (.ddl.gz or .tsv.gz).
out=<file>
Output combined DDL file. Required. Typically gzip-compressed (.ddl.gz).
overwrite=f
Set to t to allow overwriting an existing output file. Default is false (will fail if output already exists).

Processing parameters

k=31
K-mer length used for validation against the DDL records. Must match the k used when the input DDL files were originally constructed.
buckets=-1
Target bucket count for condensing. Input DDL records with more buckets than this value will be condensed by folding (taking the maximum value per destination bucket). Set to -1 to disable condensing and preserve original bucket counts.
merge=t
Master merge switch. When true, enables merging for all categories (main genome, mitochondrial, plastid, and plasmid). Setting merge=f disables all merging; each input record will appear separately in the output. Equivalent to setting mergemito, mergeplastid, mergeplasmid, and the main-genome merge flag all at once.
mergemito=t
Merge mitochondrial records (filenames containing "mito") into host genome records sharing the same TID. Default true.
mergeplastid=t
Merge plastid records (filenames containing "plastid") into host genome records sharing the same TID. Default true.
mergeplasmid=t
Merge plasmid records (filenames containing "plasmid") into host genome records sharing the same TID. Default true.
blacklist=<file>
Optional blacklist file. If specified, its header is embedded in the output DDL file for downstream tools to use during classification.
verbose=f
Print per-file loading statistics (record count and detected category) to stderr. Useful for verifying that category detection is working correctly.

Java Parameters

-Xmx4g
Default maximum heap size. Increase if processing very large numbers of DDL files or files with many records.
-Xms400m
Default initial heap size.
-eoom
Exit if an out-of-memory exception occurs. Requires Java 8u92+.
-da
Disable assertions.

Examples

Merge Two DDL Databases

# Combine bacteria and archaea DDL databases into one
ddlmerger.sh in=bacteria.ddl.gz,archaea.ddl.gz out=combined.ddl.gz

Records with matching TIDs from both files will be merged. Organisms appearing in both files will have their DDL sketches combined, producing a single merged record per TID.

Merge Host Genomes with Organelle Databases

# Merge main genomes with separate mitochondrial and plastid databases
ddlmerger.sh \
    in=vertebrates.ddl.gz,vertebrates_mito.ddl.gz,plants.ddl.gz,plants_plastid.ddl.gz \
    out=vertebrates_plants_combined.ddl.gz

The filenames vertebrates_mito.ddl.gz and plants_plastid.ddl.gz trigger automatic mitochondrial/plastid detection. Each organelle record will be merged into the matching host genome record based on shared TID, producing a single combined record per organism.

Condense to Fewer Buckets

# Reduce a high-resolution DDL database to 2048 buckets for faster classification
ddlmerger.sh in=big.ddl.gz out=small.ddl.gz buckets=2048 merge=f

Use merge=f when you only want to condense bucket counts without merging records by TID. Condensing uses max-per-bucket folding, which preserves cardinality accuracy while reducing file size and classification time.

Merge Multiple Databases Without Organelle Merging

# Combine databases but keep mitochondrial records separate
ddlmerger.sh in=host.ddl.gz,mito.ddl.gz out=combined.ddl.gz \
    mergemito=f

Setting mergemito=f keeps mitochondrial records as separate entries in the output, even if they share a TID with a main genome record. Useful when you want to classify mitochondrial and nuclear reads separately downstream.

Verbose Mode for Debugging Category Detection

# Check how files are being categorized
ddlmerger.sh in=a.ddl.gz,b_mito.ddl.gz,c_plastid.ddl.gz out=merged.ddl.gz \
    verbose=t

With verbose=t, DDLMerger prints the detected category (main, mitochondrion, plastid, plasmid) and record count for each input file to stderr, making it easy to confirm that filename-based category detection is working as expected.

Algorithm Details

Processing Pipeline

DDLMerger processes inputs in four stages:

  1. Load: All records from all input files are loaded into memory. Each record is tagged with its source category (main/mito/plastid/plasmid) based on filename detection. If buckets is set and a record has more buckets than the target, it is condensed immediately during loading.
  2. Merge: Records eligible for merging (based on their category flags) are grouped by TID. When multiple records share a TID, their DDL sketches are added together and their base/contig counts are accumulated. GC content is recomputed as a weighted average using base counts. Records whose category has merging disabled remain separate.
  3. Sort and renumber: All output records are sorted by TID, then by name, then by cardinality. After sorting, sequential IDs are reassigned starting from 0.
  4. Write: The combined, sorted, renumbered records are written to the output file in DDL format, with an optional blacklist header embedded if blacklist= was specified.

DDL Sketch Addition

When two DDL records are merged, their underlying DynamicDemiLog sketches are combined by adding bucket values together. This is a lossless operation: the combined sketch accurately represents the union of the k-mer sets from both records. The resulting cardinality estimate reflects the total unique k-mer content of the merged genome, organelle, and plasmid sequences.

Bucket Condensing

Condensing reduces a DDL sketch from a high bucket count to a lower one by folding: each destination bucket receives the maximum value from the corresponding source buckets. This operation is lossy (some cardinality precision is lost), but the max-per-bucket approach is conservative and tends to slightly overestimate rather than underestimate cardinality. Condensing is applied before merging.

Related Tools

Support

For questions and support: