ReduceColumns
Extracts specific columns from a tab-delimited vector file, producing a new file with only the selected columns. Useful for removing input dimensions from ML training data (e.g., dropping codon one-hots to test whether they contribute signal). Writes a new #dims header automatically. Processes files line-by-line with low memory overhead.
Basic Usage
reducecolumns.sh <in> <out> <col0> <col1> <col2> ...This tool extracts specified columns from a tab-delimited input file and writes them to an output file. Column numbers are zero-indexed.
Parameters
This tool uses positional arguments rather than named parameters:
Parameters
- in=<file>
- Input tab-delimited vector file (with #dims header).
- out=<file>
- Output file with only the selected columns.
- col0,col1,...
- Zero-based column indices to keep, in order.
The LAST column listed should be the target/output column.
All other columns are treated as inputs.
Supports three formats:
- 5 – Single column (0-indexed).
- 0-8 – Range: columns 0 through 8 inclusive.
- 17+ – Open range: column 17 through the last column.
Memory Parameters
- -Xmx
- Maximum heap memory (default: 2g). Automatically calculated based on available system memory
- -Xms
- Initial heap memory (default: 2g). Set to match -Xmx for consistent performance
Examples
Extract First Three Columns
reducecolumns.sh input.txt output.txt 0 1 2Extracts columns 0, 1, and 2 from input.txt and writes them to output.txt
Extract Non-Sequential Columns
reducecolumns.sh data.tsv subset.tsv 0 3 7 12Extracts columns 0, 3, 7, and 12 from data.tsv, maintaining their order in the output
Single Column Extraction
reducecolumns.sh matrix.txt column5.txt 5Extracts only column 5 from matrix.txt and saves it to column5.txt
Algorithm Details
ReduceColumns implements column extraction using ByteFile and ByteStreamWriter classes for file I/O with LineParser1 for tab-delimited parsing:
Processing Strategy
- Line-by-line parsing: Uses LineParser1 class with tab delimiter ('\t') for parsing each input line
- Column indexing: Maintains IntList data structure storing target column indices for array-based access
- Dimension preservation: Writes "#dims [columns.size()-1] 1" header to output file before processing data lines
- Memory management: Processes files via ListNum<byte[]> chunks using ByteFile.nextList() iteration
File Format Handling
- Tab-delimited input: LineParser1 splits input lines on tab characters for column extraction
- Header preservation: Skips lines starting with '#' character during data processing
- Dimension metadata: Outputs dimension count as "#dims [num_selected_columns-1] 1" format
- Order preservation: Iterates through IntList maintaining command-line argument order
Performance Characteristics
- Linear complexity: O(n × m) where n is number of rows processed and m is selected column count
- Chunk-based processing: Uses ListNum<byte[]> for processing lines in batches rather than individual line reads
- I/O implementation: Uses ByteStreamWriter with print() and tab() methods for output generation
- File handling: ByteFile class handles input file reading with automatic resource management via poisonAndWait()
Use Cases
- Feature selection: Reducing dimensionality in machine learning datasets
- Data preprocessing: Extracting relevant columns for downstream analysis
- Format conversion: Reformatting data files for specific tools or pipelines
- Dataset subsetting: Creating focused datasets from larger data matrices
Input/Output Format
Input Requirements
- Tab-delimited text file
- Consistent number of columns per row
- Optional dimension metadata in header lines starting with '#'
Output Format
- Tab-delimited text with selected columns
- Updated dimension metadata: "#dims [num_selected_columns-1] 1"
- Preserved comment lines and headers from input
- Same row order as input file
Technical Notes
Column Indexing
Columns are zero-indexed, meaning the first column is 0, second column is 1, etc. This follows standard programming conventions.
Error Handling
- Invalid column numbers will cause the program to fail
- Column numbers greater than available columns will result in errors
- Ensure input file is properly tab-delimited
Memory Management
Default memory allocation is 2GB, automatically adjusted based on system availability. For very large files, memory usage remains constant due to streaming processing.
Support
For questions and support:
- Email: bbushnell@lbl.gov
- Documentation: bbmap.org