DemuxServer

Basic Usage

demuxserver.sh port=<number>

DemuxServer implements a multi-threaded HTTP server using Java's HttpServer framework for processing barcode demultiplexing requests. It creates probabilistic PCR matrices using PCRMatrixProb instances to perform maximum likelihood barcode assignment through expectation-maximization algorithms.

Parameters

Parameters control server configuration and behavior. The server supports both basic parameters and additional configuration options accessible through the Java source code.

Basic Parameters

port=3068: Port number for the server to listen on. Default is 3068.
domain=: Domain to be displayed in the help message. Default is demux.jgi.doe.gov.

Advanced Server Parameters

verbose=false: Enable verbose output for debugging server operations.
verbose2=false: Enable additional verbose output including detailed request information.
html=false: Use HTML formatting for responses.
https=false: Enable HTTPS server mode instead of HTTP.
http=true: Use HTTP server mode (opposite of https flag).
kill=null: Kill code for remotely shutting down the server.
killcode=null: Alternative parameter name for the kill code.
printip=true: Print client IP addresses in server logs.
printheaders=false: Print HTTP headers in server logs for debugging.
countqueries=true: Enable query counting and statistics tracking.
allowlocalhost=false: Allow localhost connections to bypass access restrictions.
addressprefix="128.": IP address prefix for local network identification (default for LBL).
matrixthreads=auto: Number of threads for PCR matrix calculations. Auto-detected by default.
devmode=true: Enable development mode for PCR matrix operations.

Java Parameters

-Xmx: This will set Java's memory usage, overriding autodetection. -Xmx20g will specify 20 gigs of RAM, and -Xmx200m will specify 200 megs. The max is typically 85% of physical memory.
-eoom: This flag will cause the process to exit if an out-of-memory exception occurs. Requires Java 8u92+.
-da: Disable assertions.

Examples

Basic Server Startup

demuxserver.sh port=3068

Starts the demux server on port 3068 with default settings.

Custom Domain and Port

demuxserver.sh port=8080 domain=mylab.university.edu

Starts the server on port 8080 with a custom domain for help messages.

Development Mode with Verbose Output

demuxserver.sh port=3068 verbose=true verbose2=true devmode=true

Starts the server with detailed logging for development and debugging.

HTTPS Server with Kill Code

demuxserver.sh port=443 https=true killcode=secret123

Starts an HTTPS server on port 443 with remote shutdown capability.

High Memory Configuration

demuxserver.sh port=3068 -Xmx32g matrixthreads=8

Starts the server with 32GB memory allocation and 8 matrix calculation threads.

Algorithm Details

Server Architecture

DemuxServer implements an HttpServer instance using java.util.concurrent.Executors.newFixedThreadPool() for request handling. Handler thread count is determined by Tools.max(2, Shared.threads()), creating a fixed-size thread pool based on available CPU cores with a minimum of 2 threads. Server initialization uses iterative binding with exponential backoff (1000ms, 2000ms, 4000ms, etc.) for up to 8 attempts if port binding fails.

Request Handling Endpoints

/ - BaseHandler implements HttpHandler with ServerTools.reply() returning "Shush" response
/demux - DemuxHandler processes barcode data using toAssignmentMap() and toPairList() methods
/stats - StatsHandler returns basicStats() formatted server performance metrics
/kill - KillHandler validates kill codes via testCode() before calling KillSwitch.killSilent()
/favicon.ico - IconHandler serves favIcon byte array loaded from Data.findPath("?favicon.ico")

Demultiplexing Process

The core demultiplexing algorithm processes barcode data through the following steps:

Data Reception - ServerTools.readStreamToList() reads POST request body into ArrayList<byte[]> chunks
Data Parsing - new DemuxData(body) constructor parses byte chunks into structured barcode information
Matrix Creation - PCRMatrix.create() instantiates PCRMatrixProb using barcode type, dd.length1, dd.length2, dd.barcodeDelimiter, dd.hdistSum parameters
Matrix Population - matrix.populateExpected(dd.expectedList) and matrix.populateSplitCodes(dd) load reference barcodes
Matrix Refinement - matrix.refine(dd.codeCounts, 4) performs 4 iterations of expectation-maximization probability refinement
Assignment Generation - matrix.makeAssignmentMap(dd.codeCounts, 4) creates HashMap<String, String> linking observed to expected barcodes
Response Encoding - writeToStream() formats results using Sketch.RAW or Sketch.A48 encoding with 8MB buffer chunks

Encoding Strategies

The server supports two output encoding modes controlled by the encoding parameter:

RAW Encoding (Sketch.RAW) - ByteBuilder appends full barcode sequences with tab().append() and nl() delimiters
A48 Encoding (Sketch.A48) - Delta compression using DemuxData.encodeACGTN() and DemuxData.appendA48() methods that encode only sequence differences between consecutive entries using base-48 representation

Access Control

The server implements IP-based access control through printQuery() method logic:

Localhost detection uses clientAddress.contains("127.0.0.1") and clientAddress.contains("/0:0:0:0:0:0:0:1:") for IPv4/IPv6
X-forwarded-for header parsing via requestHeaders.getFirst("X-forwarded-for") for proxy detection
Address prefix matching compares clientAddress.startsWith(addressPrefix) for local network identification
allowLocalHost boolean parameter controls localhost access when X-forwarded-for is null

Performance Monitoring

Built-in statistics tracking uses AtomicLong counters and measurement arrays:

Query counters: queries.incrementAndGet(), internalQueries, iconQueries, baseQueries, malformedQueries AtomicLong instances
Response time tracking: elapsedTimeRemote.addAndGet(elapsed), timeMeasurementsRemote.incrementAndGet(), lastTimeRemote.set(elapsed)
Bandwidth monitoring: bytesIn.addAndGet(), bytesOut.addAndGet() atomic counters
Statistics endpoint via basicStats() method formatting counters with Tools.format("%.3f ms", avgTimeDR)

Memory Management

The server is designed for high-throughput operation with careful memory management:

Default memory allocation of 24GB (-Xmx24g, -Xms24g)
Streaming output using 8MB buffer chunks to prevent memory buildup
Automatic resource cleanup for HTTP exchanges
Out-of-memory protection with -eoom flag

Development Features

When devMode is enabled, additional development features are activated in the PCR matrix calculations, providing enhanced debugging capabilities for barcode assignment algorithms.

Server Management

Starting the Server

The server attempts iterative initialization with exponential backoff if port binding fails. It will retry up to 8 times with increasing delays (1s, 2s, 4s, etc.) to handle temporary port conflicts.

Monitoring Server Health

Use the /stats endpoint to monitor server performance:

curl http://localhost:3068/stats

Remote Server Shutdown

If a kill code was specified, the server can be shut down remotely:

curl -X POST http://localhost:3068/kill/your_kill_code

Troubleshooting

Enable verbose=true for basic request logging
Enable verbose2=true for detailed HTTP header information
Check server logs for binding errors or memory issues
Use printheaders=true to debug HTTP communication problems

Support

For questions and support:

Email: bbushnell@lbl.gov
Documentation: bbmap.org