VectorUtils

Script: vectorutils.sh Package: ml Class: VectorUtils.java

Manipulates neural-network training-vector files in tab-delimited '#dims'-header format. Combines multiple input files and generates one or more partitioned outputs, applying deduplication, subsampling, shuffling, partitioning, and positive/negative class balancing. Preserves the '#dims' header correctly—unlike plain 'shuf', which corrupts it—and enforces proper header declaration for weighted or multi-output datasets.

Basic Usage

vectorutils.sh in=<files> out=<file[:frac],...> [flags]

VectorUtils reads one or more tab-delimited input files and writes partitioned output files, applying transformations in the following order:

  1. Deduplicate (if enabled)
  2. Subsample (if samplerate < 1)
  3. Shuffle (if enabled)
  4. Partition by fraction (distribute rows across output files)
  5. Balance classes (if enabled)

Header Handling

  • The '#dims' header line is preserved as the first line of each output file.
  • If no input carries a '#dims' header and dims= is not specified, a warning is printed and no header is written. This protects against silently misinterpreting weight columns as inputs.
  • Use dims= to explicitly declare weighted or multi-output datasets, which cannot be inferred from data alone.

Memory and Threading

VectorUtils holds all rows in RAM. Set max memory with -Xmx flag based on input data size (e.g., -Xmx200g for large datasets):

vectorutils.sh -Xmx200g in=huge_dataset.tsv out=train.tsv:0.9,test.tsv:0.1 shuffle balance=0.3

Memory is automatically detected if -Xmx is unset.

Parameters

Input/Output Parameters

in=<files>
Input vector files. Accepts comma-separated list (in=a.tsv,b.tsv,c.tsv) or bare filenames (vectorutils.sh a.tsv b.tsv c.tsv). Required.
out=<f1:0.9,f2:0.1>
Output files with partition fractions. Format: filename:fraction. Fractions are normalized to sum to 1. A single out= with no fraction gets the entire dataset. Required.
dims=28,1,1
Write an explicit '#dims' header, overriding any input headers. Format: comma-separated integers. Required when the dataset is weighted (third field = 1) or has multiple outputs, since these cannot be inferred from data. Default: inherit from input or leave unset if no input header exists.
overwrite=t
(ow) Overwrite existing output files. Default: true.

Data Processing Parameters

deduplicate
(dedupe) Remove exact-duplicate rows (sort-based comparison). Default: false (off).
samplerate=1
Keep a random fraction of rows (subsample). Range: (0, 1]. A value of 0.5 keeps 50% of rows randomly selected. Default: 1 (keep all rows).
shuffle
Randomly shuffle rows (without shuffling, partition produces sequential subsets). Needed for meaningful random partitioning. Default: false (off).
balance=0
Upsample the minority class to a target fraction of each output partition's total. The label is determined by the last column (positive if >= 0.5, negative if < 0.5). Applied independently to each output after partitioning. Range: [0, 1). Typical values: 0.25–0.3. Default: 0 (off, no balancing).
seed=1
RNG seed for shuffle and subsample. Use ≥0 for deterministic reproducibility, or -1 for random seed. Default: 1.

Java Parameters

-Xmx
Set maximum heap memory (e.g., -Xmx200g). Vectors are held in RAM, so size to the dataset. Autodetected if unset. Default: autodetect.
-eoom
Exit on out-of-memory (throw RuntimeException instead of hanging). Default: off.
-da
Disable assertions (not recommended for debugging). Default: assertions enabled.

Usage Examples

Basic Partitioning

Split an input file into 80% training and 20% validation sets:

vectorutils.sh in=data.tsv out=train.tsv:0.8,valid.tsv:0.2 shuffle

The shuffle flag ensures random distribution across partitions rather than sequential splitting.

Combine Multiple Files with Subsampling and Balancing

Combine three input files, subsample to 50%, shuffle, partition into three outputs, and balance the minority class to 30% in each output:

vectorutils.sh in=a.tsv,b.tsv,c.tsv out=training.tsv:0.5,validation.tsv:0.3,test.tsv:0.2 shuffle samplerate=0.5 balance=0.3

Processing order: combine inputs → subsample 50% → shuffle → partition (50%, 30%, 20%) → balance each output independently.

Deduplication with Custom Header

Remove exact-duplicate rows from a weighted dataset and declare the layout explicitly:

vectorutils.sh in=raw_vectors.tsv out=clean_vectors.tsv deduplicate dims=28,1,1 -Xmx64g

The dims=28,1,1 header declares 28 input features, 1 output, and 1 weight column. Deduplication uses lexicographic byte-array comparison.

Reproducible Shuffle with Deterministic Seed

Shuffle and partition with a fixed seed for reproducible results across runs:

vectorutils.sh in=vectors.tsv out=fold1.tsv:0.5,fold2.tsv:0.5 shuffle seed=42

Using seed=42 ensures the same shuffle and partition structure every time the command is run.

How It Works

Header Preservation

VectorUtils treats lines beginning with '#' as headers and skips them during row processing. The '#dims' header (if present) is captured from the first input that carries one and written as the first line of each output file. This is critical because standard 'shuf' corrupts the header by treating it as a regular row.

Deduplication

Deduplication uses an in-memory sort (lexicographic byte-array order) followed by a single pass to remove consecutive duplicates. Rows are compared byte-by-byte, then by length if prefixes match.

Partitioning

After shuffle (if enabled), rows are partitioned by normalized fractions. If two outputs specify 0.9 and 0.1, the 0.1 is normalized so each receives exactly its fraction of the total. The last partition captures any remaining rows to ensure no rows are lost.

Class Balancing

Balance is applied to each output independently after partitioning. The label is the last tab-delimited column: positive if ≥0.5, negative if <0.5. The minority class is upsampled by randomly duplicating rows until it reaches the target fraction:

target_minority_count = (balance × majority_count) / (1 - balance)

For example, balance=0.3 with 70 majority and 30 minority rows results in: target = (0.3 × 70) / 0.7 = 30, so no change. With 70 majority and 20 minority: target = (0.3 × 70) / 0.7 ≈ 30, so 10 duplicates are added.

Processing Order

Operations are applied in this fixed sequence:

  1. Read all inputs: Capture the first '#dims' header; collect body rows only.
  2. Deduplicate: If enabled, sort rows and remove consecutive duplicates.
  3. Subsample: If samplerate < 1, randomly keep that fraction of rows.
  4. Shuffle: If enabled, randomize row order using the specified seed.
  5. Partition: Split rows into outputs by normalized fractions.
  6. Balance: For each output independently, upsample the minority class if enabled.
  7. Re-shuffle partitions: If balance is enabled and shuffle is enabled, re-shuffle each partition to spread duplicates.
  8. Write outputs: Write '#dims' header (if present) and rows to each output file.

Important Notes

Missing Headers

If no input file carries a '#dims' header and dims= is not provided, VectorUtils will:

This conservative approach prevents silently misinterpreting weight columns. Always use dims= when the dataset is weighted or has multiple outputs.

Weighted Datasets

If your dataset has a weight column (third field = 1 in the '#dims' header), you MUST declare dims= explicitly. For example, a 28-input, 1-output, 1-weight dataset: dims=28,1,1.

Large Datasets

All rows are held in RAM. For multi-gigabyte datasets, set -Xmx appropriately. Use -eoom to exit cleanly if memory is exhausted.

Reproducibility

Use seed=<N> (where N ≥ 0) to ensure reproducible shuffle and partition results. The default seed is 1, so results are reproducible by default.

Support

For questions and support: