DiskBench

Basic Usage

diskbench.sh path=<path> data=<8g> passes=<2> threads=<>

Benchmarks disk performance by writing and reading data files in the specified path using multiple threads. The tool measures throughput in MB/s for different I/O patterns.

Parameters

Parameters control the benchmark configuration including data size, thread count, and I/O modes. All parameters from the shell script are documented below.

Core Parameters

path=

Location to read and write test files. Directory will be created if it doesn't exist. Required parameter.

data=8g

Number of bytes to process per pass. Accepts size suffixes (k, m, g). Default is 8 gigabytes. This amount is divided among threads.

threads=

Number of threads to use. By default, uses all logical threads available. In RW mode the number of active threads is doubled (separate read and write threads).

passes=2

Number of benchmark passes to perform after the initial write. Default is 2 passes.

mode=rw

I/O mode for testing:

r: Test read speed only
w: Test write speed only
rw: Test read and write speed simultaneously (default)

Advanced Parameters

lines=: Maximum number of lines to process. Default is unlimited (Long.MAX_VALUE).
verbose=false: Enable verbose output for detailed benchmark information.
gzip=false: Enable gzip compression for test files (note: parameter name maps to verbose internally in current implementation).
read=false: Shortcut to set mode to read-only (equivalent to mode=r).
write=false: Shortcut to set mode to write-only (equivalent to mode=w).
readwrite=false: Shortcut to set mode to simultaneous read/write (equivalent to mode=rw).
printtid=false: Print thread IDs during benchmark execution for debugging.
processbis=false: Enable line processing when using BufferedInputStream methods (splits data into lines).
preread=false: Perform a preliminary read pass before the main benchmark to warm up the disk cache.

I/O Method Parameters

method=

Select the I/O method for reading files. Options:

BYTEFILE or bf: Use BBTools ByteFile (default)
TEXTFILE or tf: Use BBTools TextFile
QUICKFILE or qf: Use BBTools QuickFile
BUFFEREDINPUTSTREAM or bis: Use Java BufferedInputStream
FILEINPUTSTREAM or fis: Use Java FileInputStream
BUFFEREDINPUTSTREAM2 or bis2: BufferedInputStream with ArrayList storage
FILEINPUTSTREAM2 or fis2: FileInputStream with ArrayList storage

bytefile=false

Use ByteFile method for I/O (equivalent to method=bf).

textfile=false

Use TextFile method for I/O (equivalent to method=tf).

quickfile=false

Use QuickFile method for I/O (equivalent to method=qf).

bufferedinputstream=false

Use BufferedInputStream method for I/O (equivalent to method=bis).

fileinputstream=false

Use FileInputStream method for I/O (equivalent to method=fis).

bufferedinputstream2=false

Use BufferedInputStream with ArrayList storage (equivalent to method=bis2).

fileinputstream2=false

Use FileInputStream with ArrayList storage (equivalent to method=fis2).

buffer=4096

Buffer size in bytes for stream-based I/O methods. Default is 4096 bytes (4KB).

bufferlen=4096

Alias for buffer parameter. Sets buffer size for stream I/O operations.

Java Parameters

-Xmx: This will set Java's memory usage, overriding autodetection. -Xmx20g will specify 20 gigs of RAM, and -Xmx200m will specify 200 megs. The max is typically 85% of physical memory.
-eoom: This flag will cause the process to exit if an out-of-memory exception occurs. Requires Java 8u92+.
-da: Disable assertions.

Examples

Basic Disk Benchmark

diskbench.sh path=/tmp/benchmark data=1g passes=3

Performs disk benchmarking in /tmp/benchmark with 1GB of data per pass, running 3 passes in read-write mode.

Read-Only Performance Test

diskbench.sh path=/data/test mode=r data=2g threads=8

Tests read performance only with 2GB of data using 8 threads. Files must already exist from a previous write test.

Write-Only Performance Test

diskbench.sh path=/scratch/bench mode=w data=5g passes=1

Tests write performance only with 5GB of data for 1 pass. Useful for testing storage write speeds.

Comparing I/O Methods

diskbench.sh path=/tmp/test method=bis data=500m
diskbench.sh path=/tmp/test method=bf data=500m

Compare performance between BufferedInputStream and ByteFile methods using the same data size.

High-Performance SSD Test

diskbench.sh path=/fast/ssd threads=16 data=10g mode=rw passes=5

Intensive benchmark for high-performance SSDs using 16 threads, 10GB data, simultaneous read/write mode, and 5 passes.

Algorithm Details

Benchmark Architecture

DiskBench implements multithreaded I/O benchmarking through WriteThread and ReadThread classes with the following components:

Thread Management Strategy

Write Threads: Each thread writes a portion of the total data size (data/threads) to separate files
Read Threads: Each thread reads from its corresponding file created during the write phase
Simultaneous Mode: In read-write mode, separate read and write threads operate concurrently, effectively doubling active thread count
Thread Synchronization: Uses Thread.join() to ensure all threads complete before measuring elapsed time

Data Generation Algorithm

Write operations implement the writeRandomData() method using ByteStreamWriter and ByteBuilder classes:

Random Data Source: Uses Shared.threadLocalRandom() for thread-local Random instances
Bit Shifting Implementation: Generates 5.33 bytes per Random.nextInt() call using 6-bit right shifts (shift=6, shiftsPerRand=32/6)
Character Encoding: Maps random bits via (byte)(33+x&63) to ASCII range 33-96
Buffer Management: Uses ByteBuilder with 66000-byte capacity, writing 1000-character blocks before newlines
Line Termination: Calls bb.nl() after every 1000-character generation loop to create line-based text files

Multiple I/O Method Support

The benchmark implements seven different I/O methods through separate run methods in the ReadThread class:

BYTEFILE: runBf() method using ByteFile.makeByteFile() with nextLine() iteration
TEXTFILE: runTf() method using TextFile constructor with String-based nextLine() processing
QUICKFILE: runQf() method using QuickFile constructor with byte array nextLine() processing
BUFFEREDINPUTSTREAM: runBis(ffin, true) method using ReadWrite.getInputStream() with buffered flag
FILEINPUTSTREAM: runBis(ffin, false) method using ReadWrite.getInputStream() without buffering
BUFFEREDINPUTSTREAM2/FILEINPUTSTREAM2: runBis2() methods with ArrayList<byte[]> storage and 800-element capacity management

Performance Measurement

Timing Implementation: Uses Timer class with t.start(), t.stop(), and t.elapsed for nanosecond-precision timing
Throughput Calculation: Computes MB/s using Tools.format("%.3f MB/s", (1000.0*data)/t.elapsed)
Phase Separation: Tracks initialWriteElapsed separately from benchmark pass timing
Overall Performance: Excludes initial write from total: (1000.0*(data*passes))/(t0.elapsed-initialWriteElapsed)

File Management

Filename Generation: makeFnames() creates names using path+pass+"_"+i+"_"+(System.nanoTime()&0xFFFF)+"_"+randy.nextInt(4096)
File Cleanup: delete() method iterates through filename arrays calling f.delete() after each pass
Directory Creation: Constructor calls pfile.mkdirs() if benchmark path doesn't exist
Overwrite Handling: FileFormat.testOutput() uses overwrite=true from parser configuration

Memory Usage Characteristics

DiskBench implements memory constraints through shell script configuration and buffer management:

Default Heap: Shell script sets z="-Xmx400m" and z2="-Xms400m" for 400MB heap allocation
Buffer Allocation: bufferlen=4096 for stream methods, ByteBuilder(66000) for write operations
ArrayList Management: runBis2() methods use ArrayList<byte[]>(800) with periodic reinitialization
Thread-Local Storage: Each ReadThread maintains individual read counters and cache arrays

Benchmarking Modes Implementation

Read Mode (mode=READ=2): process() calls read(fnamesR) using existing files from initial write
Write Mode (mode=WRITE=3): process() calls delete(fnamesR), write(fnamesW) creating new files each pass
Read-Write Mode (mode=READWRITE=1): process() calls readWrite(fnamesW, fnamesR) with concurrent thread execution

Performance Implementation Details

Thread Synchronization: Uses Thread.join() in nested loops checking getState()!=Thread.State.TERMINATED
Data Size Distribution: Each thread processes data/threads bytes, with WriteThread(fname, size) constructor
Buffer Size Control: Configurable bufferlen parameter affects InputStream.read(buffer) performance
ByteFile Optimization: Sets ByteFile.FORCE_MODE_BF1=true for consistent I/O behavior

Support

For questions and support:

Email: bbushnell@lbl.gov
Documentation: bbmap.org