Tadpipe

Basic Usage

tadpipe.sh in=reads.fq out=contigs.fa

TadPipe assembles preprocessed paired reads through a fixed sequence of stages, each run as a BBTools launcher in its own JVM: deduplication (Clumpify), overlap correction (BBMerge), Clumpify correction, Tadpole kmer correction at k=62, overlap merging with REM passes (BBMerge), residual quality trimming (BBDuk), read extension (Tadpole), and ordered multi-K assembly (Tadpole). Inputs must already be adapter-trimmed, filtered, and quality-score calibrated. TadPipe performs no adapter trimming or contaminant filtering.

Parameters

Parameters are grouped into basic parameters, stage switches, and final assembly controls. Phase-specific options are passed to individual tools in the pipeline by using the appropriate prefix.

Basic Parameters

in=<file>: Paired input reads; interleaved when in2 is omitted. Required. Alias: in1.
in2=<file>: Optional read 2, if reads are in two files. Inputs must be named readable files; stdin is not supported.
out=contigs.fa: Output file name for the final assembly. Default: contigs.fa. The file is published only after every stage succeeds. An existing output stops the run unless overwrite=t.
temp=<directory>: Parent directory for a unique work directory. Default: Java tmpdir.
delete=t: Delete pipeline FASTQ intermediates only after successful assembly. Commands and per-stage logs are always retained, and all intermediates are retained on failure. Default: true.
gz=t: Compress intermediates through native BBTools I/O (.fq.gz files). Default: true.
overwrite=f: Allow replacement of an existing final output. Default: false.
dryrun=f: Validate paths and print/save commands without running them. The commands are written to commands.sh in the work directory. Default: false.
t=<auto>: Threads per child. Stages run serially to reuse the heap budget and avoid keeping two kmer tables in memory. Default: auto-detected.
-Xmx16g: Maximum heap per child stage. The coordinator uses only 256 MB. Default: 14g.
childheap=<auto>: Alternative explicit child heap, for example childheap=16g. Default: 14g.
zl=4: Intermediate compression level, 1 to 9. Default: 4.

Stage Switches

All stage switches default to true.

dedupe=t: Sequence-based paired deduplication, not optical-only. Both mates must match under Clumpify's duplicate criteria.
ecco=t: BBMerge overlap correction.
clump=t: Clumpify correction of both mates, with pairing restored.
ecc=t: Tadpole kmer correction at K62.
merge=t: Overlap merging, then optional REM passes.
rem=t: REM at K124, K145, and K93 with accumulated merged-read evidence. Applies only when merge=t.
qtrim=t: Quality-trim and filter the residual unmerged reads.
extend=t: Extend merged and unmerged reads, with both sets as evidence.
nn=t: Use the bundled fusion neural network at cutoff 0.667098.

Final Assembly Controls

These apply only to the final assembly, never to correction, merging, or extension.

k=124,300,96,64,32: Ordered final assembly/traversal schedule. Repeats are preserved. Default: 124,300,96,64,32.
assemblek=124: Initial contigging K for the final assembly. Default: 124.
graphk=<last K>: Final graph K. Defaults to the last entry in k.
bridgek=<auto>: Override the eligible bridging K list.
fusek=<auto>: Override the eligible fusion K list.

These are aliases for assemble_k, assemble_assemblek, assemble_graphk, assemble_bridgek, and assemble_fusek. The last occurrence wins across aliases. Changing k alone keeps assemblek=124; set both to change the initial K.

Coverage guards are set to 1.75 (fusecoverageratio and graphmergecoverageratio). They can reject genuine joins across uneven coverage. Set assemble_fusecoverageratio=0 and/or assemble_graphmergecoverageratio=0 to disable them independently. For reads lacking long merged or extended support, use k=124,96,64,32.

Phase-Specific Parameter Examples

Parameters can be passed to individual phases by prefixing them with the phase name. Here are examples of how to use phase-specific parameters:

assemble_k=124,96,64,32: Set the ORDERED assembly/traversal schedule for the final assembly. Same as the k= alias.
assemble_assemblek=124: Set the initial contigging K for the final assembly. Also accepts assemblek=124.
assemble_mincontig=500: Override the final assembly minimum contig length.
assemble_mcs=2: Override the final assembly minimum seed kmer count.
assemble_mce=1: Override the final assembly minimum extension kmer count.
assemble_prefilter=t: Enable the final assembly prefilter.
dedupe_subs=0: Require exact duplicates in deduplication. The native default permits 2.
clump_passes=6: Set Clumpify correction passes (pipeline default 4).
merge_strict=t: Apply strict mode to all merging passes, not to overlap correction (ecco).
merge145_extend2=100: Override only the K145 REM pass extension.
extend_el=20: Set left extension for both read sets.
extendu_k=96: Override only the unmerged-read extension K.

Valid Phase Prefixes

Use these prefixes to pass parameters to specific phases of the pipeline:

dedupe_: Paired deduplication (Clumpify) stage. Defaults: dedupe=t, optical=f, ecc=f, passes=1, unpair=f, repair=t.
ecco_: Overlap correction stage (BBMerge in ecco mode). Defaults: ecco=t, mix=t, kfilter=1, k=31, ordered=t.
clump_: Clumpify correction stage. Alias: clumpify_. Defaults: ecc=t, dedupe=f, passes=4, unpair=t, repair=t.
ecc_: Tadpole kmer correction stage. Alias: correct_. Defaults: ecc=t, k=62, wash=t, tu=t, tossdepth=1, ldf=0.15, ordered=t. Use ecc_k to change the K explicitly.
merge_: Applies to all merge passes (merge0 through merge93), including REM passes. Defaults for each pass: interleaved=t, ordered=t. This group is applied after each pass's own defaults, so merge_k would override k in every REM pass; use merge124_k, merge145_k, or merge93_k for pass-specific K values.
merge0_: First (plain overlap) merge pass only. Default: kfilter=1.
merge124_: REM pass at K124 only. Defaults: rem=t, k=124, extend2=120.
merge145_: REM pass at K145 only. Defaults: rem=t, k=145, extend2=140.
merge93_: REM pass at K93 only. Defaults: rem=t, k=93, extend2=100, strict=t.
qtrim_: Residual quality-trim stage (BBDuk). Defaults: qtrim=r, trimq=15, maq=14, minlen=90, maxns=1, ordered=t.
extend_: Read extension stages for both read sets. Alias: extend1_. Defaults: mode=extend, mce=4, el=10, ordered=t; er=10 for merged reads and er=0 for unmerged reads.
extendm_: Merged-read extension only. Defaults: k=145, er=10.
extendu_: Unmerged-read extension only. Defaults: k=124, er=0.
assemble_: Final ordered multi-K assembly (Tadpole). Defaults: assemblek=124, k=124,300,96,64,32, minprob=0, minprobmain=f, fusecoverageratio=1.75, graphmergecoverageratio=1.75. With nn=t, also fusenet (networks/tadpole_fusion.bbnet) and fusencutoff=0.667098.

Stage-specific overrides follow group-wide overrides. Prefixes that name pipeline files or pairing (in, out, extra, config, append, overwrite, interleaved, int, gfa, dot) are rejected. The prefixes filter_, trim_, and extend2_ are not valid in this pipeline. Unknown options are rejected by the receiving tool, not silently ignored.

Examples

Basic Assembly

tadpipe.sh in=reads.fq out=contigs.fa

Basic usage with paired-end reads in a single interleaved file. Uses default parameters for all phases.

Separate Paired Files

tadpipe.sh in=reads_R1.fq in2=reads_R2.fq out=assembly.fa

Assembly from separate paired-end read files (R1 and R2).

Custom Final Assembly Schedule

tadpipe.sh in=reads.fq out=contigs.fa k=96,124,300,64,32 assemblek=96

Sets a different ordered final K schedule and initial contigging K. Both are set because changing k alone keeps assemblek=124.

Custom Merge and Extension

tadpipe.sh in=reads.fq out=contigs.fa merge_strict=t merge145_extend2=100 extend_el=20

Applies strict mode to all merge passes, shortens the K145 REM pass extension, and sets left extension to 20 for both read sets.

Retain Intermediate Files

tadpipe.sh in=reads.fq out=contigs.fa delete=f temp=/scratch/assembly_temp/

Keeps FASTQ intermediates after success and uses a custom parent for the work directory. Commands and per-stage logs are retained either way.

Explicit Heap and Threads

tadpipe.sh in=reads.fq out=contigs.fa -Xmx16g t=8 temp=/fast_storage/tmp/

Sets a 16 GB heap for each child stage and 8 threads per child, with the work directory on fast storage. Stages run one at a time, so only one child heap is in use at once.

Dry Run

tadpipe.sh in=reads.fq out=contigs.fa dryrun=t

Validates paths and writes the planned stage commands to commands.sh in the work directory without launching any stage.

Algorithm Details

Pipeline Overview

TadPipe runs a fixed sequence of stages. Each stage is a BBTools launcher invocation in its own JVM, and stages run one at a time. The stage switches (dedupe, ecco, clump, ecc, merge, rem, qtrim, extend) can disable stages; the final assembly always runs. With default settings the pipeline launches 13 child processes: dedupe, ecco, clump, ecc, four merge passes, combine, qtrim, two extensions, and assembly.

Phase 1: Paired Deduplication

With dedupe=t (default), Clumpify removes duplicate read pairs:

Sequence-based, not optical-only; both mates must match under Clumpify's duplicate criteria
Defaults: optical=f, ecc=f, passes=1, unpair=f, repair=t
Native default permits 2; dedupe_subs=0 requires exact duplicates

With dedupe=f, reformat.sh pairs the input reads instead.

Phase 2: Overlap Correction (ECCO)

BBMerge runs in ecco mode with ecco=t, mix=t, kfilter=1, k=31, and ordered=t. The help text describes this stage as BBMerge overlap correction.

Phase 3: Clumpify Correction

Clumpify corrects both mates and restores pairing, with ecc=t, dedupe=f, passes=4 (clump_passes overrides this count), unpair=t, and repair=t.

Phase 4: Tadpole Kmer Correction

Tadpole runs error correction at k=62 with ecc=t, wash=t, tu=t, tossdepth=1, ldf=0.15, and ordered=t. The Tadpole stages also set prealloc=f, prefilter=f, and hashkmers=explicit.

Phase 5: Overlap Merging and REM

BBMerge runs up to four merge passes in this order:

merge0: plain overlap merge, kfilter=1 (always runs)
merge124: REM at k=124, extend2=120 (rem=t only)
merge145: REM at k=145, extend2=140 (rem=t only)
merge93: REM at k=93, extend2=100, strict=t (rem=t only)

Each pass takes the unmerged output of the previous pass as input and receives the merged output of all earlier passes as extra evidence. The merged outputs are then concatenated by cat.sh (06_combine) when more than one pass ran. All merge passes default to interleaved=t and ordered=t.

Phase 6: Residual Quality Trimming

With qtrim=t, BBDuk quality-trims and filters the residual unmerged reads using qtrim=r, trimq=15, maq=14, minlen=90, maxns=1, and ordered=t. No adapter trimming is performed; adapter trimming and contaminant filtering are not part of this pipeline.

Phase 7: Read Extension

With extend=t, Tadpole extends the merged and unmerged read sets. Each set is passed as extra evidence to the other:

Merged reads (extendm, runs only when merge=t): mode=extend, k=145, el=10, er=10, mce=4
Unmerged reads (extendu): mode=extend, k=124, el=10, er=0, mce=4

Phase 8: Ordered Multi-K Assembly

Tadpole performs the final assembly with the ordered schedule k=124,300,96,64,32, with repeats preserved. Defaults are assemblek=124, graphk equal to the last entry of k, minprob=0, minprobmain=f, fusecoverageratio=1.75, and graphmergecoverageratio=1.75. With nn=t, the fusion network networks/tadpole_fusion.bbnet is used at fusencutoff=0.667098. When merging is disabled, the reads are interleaved input. When merging is enabled, the merged reads are the main input and the unmerged reads are passed as extra input.

Performance Characteristics

The pipeline resource requirements:

Memory Usage: Each child stage receives the childheap value, default 14g. The coordinator uses 256 MB. Only one child runs at a time.
Disk Usage: Intermediates are compressed through BBTools I/O by default (gz=t). With delete=t (default), FASTQ intermediates are removed after a successful assembly; commands and logs are always kept.
Processing Time: Not documented in the source or help text. The coordinator prints "Finished <stage> in N seconds" after each stage.
Output Quality: The default final schedule reaches k=300 (k=124,300,96,64,32).

Algorithm Features

Long Kmer Support: The default final schedule uses k up to 300 (k=124,300,96,64,32)
Correction Stages: BBMerge overlap correction (k=31), Clumpify correction, and Tadpole kmer correction at k=62
Paired Deduplication: Clumpify deduplication requires both mates to match
REM Merging: REM passes at k=124, 145, and 93 use accumulated merged-read evidence
Coverage Guards: Fusion and graph-merge coverage ratios are 1.75; set assemble_fusecoverageratio=0 and/or assemble_graphmergecoverageratio=0 to disable them
Ordered Assembly: The final assembly traverses the k schedule in the order given

Implementation Strategy

Work Directory: Creates a unique work directory under temp, writes commands.sh listing every stage command, and writes a log for each stage
Sequential Processing: Each stage must create its declared outputs before the next stage starts
Publication: The final assembly is moved to out= only after every stage succeeds and the file contains sequence
Automatic Cleanup: With delete=t, pipeline FASTQ intermediates are removed after success
Parameter Forwarding: Group-wide overrides are applied first, then stage-specific overrides, so the stage-specific value wins
Launch: Each stage runs its BBTools launcher through bash with an explicit argument list

Support

For questions and support:

Email: bbushnell@lbl.gov
Documentation: bbmap.org