RegressionTrainer

Overview

RegressionTrainer creates neural networks for continuous-valued prediction tasks, differing fundamentally from Train (which handles boolean classification). It implements pure mean-squared-error regression with automatic input standardization, mini-batch training, and support for multi-output networks.

Training data is streamed into per-sample float arrays, so memory scales with sample count rather than file size and is not limited by Java's single-array length limit. After writing, the network is reloaded and verified against the in-memory model to catch serialization errors.

The tool supports continuing training from existing networks (netin=), multiple output layers, configurable hidden activations, structured sparsity with density control, and optional SIMD-accelerated training with multi-threaded gradient accumulation.

Basic Usage

regressiontrainer.sh in=training.tsv out=net.bbnet dims=48,64,32,1 epochs=100

At minimum, supply training data (in=), output filename (out=), network architecture (dims=), and optionally epochs. The dims parameter specifies layer widths: first value is input width, middle values are hidden layers, last value is output width.

Input Format

Input files must be tab-delimited text with a required '#dims <inputs> <outputs>' header line, followed by data rows. Each row contains input columns followed by target (label) columns as floats:

#dims 48 1
0.123	0.456	... (48 inputs)	0.789 (1 output)
0.234	0.567	... (48 inputs)	0.890 (1 output)

Memory and Scaling

Memory scales roughly as: rows × 4 bytes × (inputs + outputs) + 40-50 bytes overhead per row + model/JVM overhead. For five million rows with 48 inputs: approximately 1.2 GB total. Use -Xmx to set heap size explicitly; do not size from observed RSS since the JVM reserves far more than it needs.

Common Use Cases

Basic Regression with Internal Validation

Train on 90% of data, automatically validate on held-out 10%:

regressiontrainer.sh in=data.tsv out=model.bbnet dims=48,64,32,1 epochs=100 vfraction=0.1

The vfraction parameter controls how much of the training file is held out for validation. Validation rows are never trained on and are excluded from normalization statistics. The best-validating weights are kept and exported.

External Validation Set

When validation data comes from a different source or time period:

regressiontrainer.sh in=train.tsv valin=holdout.tsv out=model.bbnet dims=48,64,32,1 epochs=100

External validation rows are never trained on and excluded from normalization, preventing data leakage. Do not combine valin= with vfraction>0 — use one or the other.

Multi-Output Network

Predict multiple continuous values simultaneously:

regressiontrainer.sh in=data.tsv out=multi.bbnet dims=48,64,4 epochs=50

The last dims entry (4) sets output width. Loss is summed squared error across all outputs. Each output uses the same final activation. The round-trip check verifies all outputs independently.

Continue Training from Existing Network

Resume training from a previously-trained network with new data:

regressiontrainer.sh in=newdata.tsv netin=base.bbnet out=retrained.bbnet epochs=50

Training runs in raw input space (no re-standardization). The saved network already has input standardization folded in; applying it again would create a double transform. dims= may be omitted when netin= is supplied.

SIMD Acceleration with Multiple Workers

Train using vectorized float kernels and multi-threaded gradient accumulation:

regressiontrainer.sh in=data.tsv out=model.bbnet dims=48,128,64,1 simd=t threads=4 epochs=100

Results are close to but not bit-identical to the scalar path (different reduction order). threads>1 requires simd=t. Each worker owns private gradient copies; Adam and final reduction remain serial. Measured 1.46-3.39× speedup depending on layer width.

Custom Hidden Activations

Mix different activation functions in hidden layers:

regressiontrainer.sh in=data.tsv out=model.bbnet dims=48,64,32,1 hidden=tanh,swish,sig epochs=100

Each hidden cell randomly draws one activation from the comma-separated list, seeded for reproducibility. Default 'tanh' keeps the hardcoded fast path and is bit-identical to historical results.

Parameters

RegressionTrainer accepts parameters in key=value format from the command line. All parameters are case-insensitive. Required parameters have no defaults; optional parameters show defaults below.

Examples

Example 1: Simple Regression with Default Parameters

regressiontrainer.sh in=house_prices.tsv out=price_model.bbnet dims=15,32,1

This trains a network with 15 input features (e.g., square footage, location, age), one hidden layer of 32 neurons, and a single output (predicted price). Training uses 60 epochs (default), 8192 mini-batch size, 0.003 learning rate, and rslog output activation. 10% of data is held out for validation automatically.

Example 2: Extended Training with Custom Learning Rate

regressiontrainer.sh in=stock_returns.tsv out=returns_model.bbnet dims=50,128,64,1 epochs=200 lr=0.001 wd=1e-3

Trains a deeper network (50 inputs → 128 → 64 → 1 output) for 200 epochs with a lower learning rate (0.001) and higher weight decay (1e-3) for regularization. Useful when overfitting is suspected or when longer training helps.

Example 3: High-Dimensional Data with Memory Limiting

regressiontrainer.sh in=image_features.tsv out=image_model.bbnet dims=1024,512,256,1 -Xmx4g epochs=50 batch=4096

Handles 1024-dimensional input (e.g., image features) with reduced batch size (4096 vs default 8192) to fit in 4GB heap. The network compresses through 512 → 256 → 1 output neuron. Epoch count is lowered to 50 since the smaller batch provides more updates per epoch.

Example 4: Multi-Output Network for Simultaneous Predictions

regressiontrainer.sh in=weather.tsv out=weather_multi.bbnet dims=100,64,32,3 valin=test_weather.tsv epochs=100

Trains a network to predict 3 outputs simultaneously (e.g., temperature, humidity, wind speed). The '#dims 100 3' header specifies 100 inputs and 3 outputs. External validation set (test_weather.tsv) is used; all metrics track all 3 outputs together under summed MSE.

Example 5: SIMD Acceleration for Large Networks

regressiontrainer.sh in=bigdata.tsv out=fast_model.bbnet dims=200,512,256,128,1 simd=t threads=4 epochs=100 -Xmx8g

Uses vectorized float kernels (simd=t) and 4 worker threads for independent gradient accumulation, achieving 2-3× wall-time speedup on this wide network. Results differ slightly from scalar training (reduction order), so verify on validation set. Requires 8GB heap for the large dimensions.

Example 6: Continue Training from Existing Network

regressiontrainer.sh in=new_data.tsv netin=base_model.bbnet out=retrained_model.bbnet epochs=30

Resumes training from a previously-trained network (base_model.bbnet) using new data. No dims= needed — the existing network supplies them. Training runs in raw input space without re-standardization (it was already folded in). The loaded network is verified before training starts.

Algorithm Details

Standardization and Export

Input features are Z-score standardized (subtract mean, divide by standard deviation) during training for stable gradient flow. The standardization transform is learned from training data and folded into the first layer's weights and biases at export. The saved network consumes raw (unstandardized) inputs, eliminating the need for upstream preprocessing.

When continuing training with netin=, the loaded network already has standardization folded in (mean=0, sd=1 makes the transform a no-op), so re-standardizing would apply a double transform. The trainer sets mean and sd to make both the forward transform and the export fold no-ops.

Optimizer: Adam with Cosine Learning-Rate Decay

Training uses the Adam optimizer (Adaptive Moment Estimation) with bias correction and cosine annealing learning-rate decay. The learning rate decays as: lr_now = lr × 0.5 × (1 + cos(π × (epoch-1) / epochs)). This starts high and smoothly decays to nearly zero by the final epoch, encouraging fine-tuning near convergence.

Adam hyperparameters are β₁=0.9 (first-moment exponential decay), β₂=0.999 (second-moment exponential decay), and ε=1e-8 (numerical stability). Gradients are accumulated over mini-batches and bias-corrected before the update step.

Mini-Batch Training with Best-Model Retention

Each epoch consists of batches of size 'batch' (default 8192). After each batch, one Adam update is applied. Validation error is measured after each epoch on the held-out set (internal or external), and the weights with lowest validation error are retained and exported. This prevents overfitting by stopping at the point of best generalization.

Output Activations

Three output activations are supported:

• linear: fin(z) = z. Unbounded, no saturation. Suitable for any target range.
• rslog: fin(z) = -log(1-z) if z < 0, else log(1+z). Unbounded, symmetric around 0. Default. Produces smooth gradients in all ranges.
• sigmoid: fin(z) = 1/(1+exp(-z)). Bounded [0,1], requires targets in [0,1]. Saturates at exactly 0 and 1, which can slow training near boundaries.

Derivatives of output activations are used during backpropagation and are computed from the activated value to minimize loss of precision.

Hidden Activations and Derivatives

Default hidden layers use tanh (hardcoded for speed). Custom activation sets are specified with hidden=name1,name2,... and each hidden cell draws one at random, seeded for reproducibility. When non-default activations are used, pre-activations are retained to compute derivatives via derivativeXFX(z, fx) because some functions (e.g., Swish) lack a simple derivative formula.

Structured Sparsity and Density Control

The density parameter (0 < density ≤ 1) controls what fraction of hidden-layer edges are kept during training. The CellNet topology generator uses this to create sparse connectivity patterns. Pruned edges remain exactly 0 throughout training and in the export. While smaller models are created, they do not train faster since zeros remain in dense arrays; benefits appear only when specialized sparse kernels are used (sparse=t, but measured slower in most cases).

Multi-Output Support

Networks can have multiple outputs (last dims entry > 1). Loss is summed squared error across all outputs: loss = Σ(output_k - target_k)². Each output uses the same final activation function. Validation and round-trip checks verify all outputs independently.

Gradient Accumulation and Thread Workers

In SIMD mode (simd=t), gradients can be accumulated by multiple worker threads (threads=N). Each worker processes a fixed portion of the batch and owns private forward/backward scratch and gradient buffers. Worker gradients are reduced serially into the master accumulator in worker order (deterministic but not commutative for floats). Adam updates remain serial. This achieves reproducible parallelism without batching effects from different batch orderings.

Round-Trip Verification

After export, the saved network is immediately reloaded via CellNetParser and tested on up to 1000 samples (or all samples if fewer). Maximum absolute difference between the written net and the in-memory model is reported. If maxDiff > 1e-3, output "(SUSPICIOUS!)" and sets errorState to prevent silent corruption from going unnoticed.

Comparison with Train

Train (ml.Trainer) is designed for boolean classification with mechanisms for class balancing, operating point selection (via cutoff), and performance metrics (FPR/FNR). RegressionTrainer, by contrast, implements pure MSE regression with no balancing or classification machinery. It is the right choice for continuous-valued prediction tasks.

Performance Tips

• Layer sizing: Start with hidden layers roughly 1.5-2× the input width and decay toward output width. For 48 inputs, try dims=48,64,32,1 before experimenting.
• Batch size: Larger batches (8192-32768) exploit CPU/SIMD efficiency but may reduce gradient signal. Smaller batches (1024-4096) add noise but escape local minima. Default 8192 balances both.
• Learning rate: If loss oscillates, divide lr by 2. If loss decreases slowly, multiply by 1.5. Default 0.003 works for most 50-100 neuron layers.
• SIMD acceleration: Use simd=t and threads=4 for networks with layers > 128 neurons. Narrow networks (< 64 neurons) see little benefit. Measure wall time; reported speedups of 1.46-3.39× assume your data layout and CPU.
• Memory management: Use -Xmx explicitly; do not rely on default sizing. For 5 million 48-input samples: -Xmx2g is comfortable with overhead.
• Validation: Always use either vfraction or valin. Watching training MSE alone is insufficient; validation error tracks generalization and drives best-model retention.

Support

For questions and support:

• Email: bbushnell@lbl.gov
• Documentation: bbmap.org
• Related tool: Train for boolean classification tasks