XCalcMem

Basic Usage

Xcalcmem.sh [java_flags] [default_memory] [memory_percentage]

This script is typically called internally by other BBTools scripts to determine appropriate Java memory settings. It can also be used standalone to check available memory.

Parameters

Xcalcmem processes Java memory flags and system parameters to calculate memory allocation.

Java Memory Flags

-Xmx[size]: Sets maximum Java heap size. Size can be specified with suffixes: g (gigabytes), m (megabytes), k (kilobytes). Example: -Xmx8g for 8 gigabytes maximum heap.
Xmx[size]: Alternative format for maximum Java heap size without leading dash. Automatically converted to -Xmx format internally.
-Xms[size]: Sets initial Java heap size. Size can be specified with suffixes: g (gigabytes), m (megabytes), k (kilobytes). Example: -Xms2g for 2 gigabytes initial heap.
Xms[size]: Alternative format for initial Java heap size without leading dash. Automatically converted to -Xms format internally.

Java Assertion Flags

-da: Disable assertions in Java. Passed through to Java runtime for debugging control.
-ea: Enable assertions in Java. Passed through to Java runtime for debugging control.

Memory Calculation Parameters

default_memory: Default memory amount to use when system memory cannot be determined. Specified in kilobytes. Default: 3200000 (approximately 3.2GB). Can include suffixes: g (gigabytes), m (megabytes), k (kilobytes).
memory_percentage: Percentage of available memory to allocate for Java processes. Default: 84 (uses 84% of available memory). Range: 1-100.

Memory Calculation Algorithm

Xcalcmem uses the freeRam function to determine Java memory settings:

Memory Detection Process

System Memory Check: Reads /proc/meminfo to determine virtual and physical memory availability
ulimit Verification: Checks user memory limits using ulimit -v
Memory Calculation: Uses formula: RAM = ((available_memory - 500MB) × percentage) / 1024
Fallback Mechanism: Uses default memory value if system memory cannot be determined

Memory Metrics Analyzed

Virtual Memory Free (vfree): CommitLimit - Committed_AS from /proc/meminfo
Physical Memory Free (pfree): MemFree + Cached + Buffers from /proc/meminfo
User Limit: Virtual memory limit from ulimit -v
Conservative Selection: Uses minimum of available memory sources for safety

Flag Processing Logic

The script implements Java flag processing using parseXmx function:

Memory Flag Synchronization

Xmx without Xms: Automatically sets Xms to same value as Xmx using substring extraction (cut -d'x' -f 2)
Xms without Xmx: Automatically sets Xmx to same value as Xms using substring extraction (cut -d's' -f 2)
Format Normalization: Converts all memory flags to standard -Xmx and -Xms format
Assertion Handling: Preserves and passes through Java assertion flags (-ea/-da)

Examples

Basic Memory Calculation

Xcalcmem.sh

Calculates available system memory using default parameters (3.2GB fallback, 84% allocation ratio).

Custom Default Memory

Xcalcmem.sh 8g

Sets fallback memory to 8GB if system memory cannot be determined automatically.

Custom Memory Percentage

Xcalcmem.sh 4g 75

Uses 4GB fallback memory and allocates 75% of available system memory instead of default 84%.

Java Flag Processing

Xcalcmem.sh -Xmx16g -ea

Processes maximum heap of 16GB, enables assertions, and automatically sets initial heap to match.

Output Variables

Xcalcmem sets several shell variables for use by calling scripts:

Global Variables Set

RAM: Calculated memory amount in megabytes available for Java processes. Used by other BBTools scripts to set appropriate -Xmx values.
z: Formatted -Xmx flag string with calculated or specified maximum heap size. Ready for use in Java command lines.
z2: Formatted -Xms flag string with calculated or specified initial heap size. Ready for use in Java command lines.
EA: Java assertion flag (-ea or -da) if specified. Passed through for debugging control.
set: Boolean flag indicating whether custom memory flags were provided (1) or system calculation was used (0).

System Compatibility

Xcalcmem includes special handling for different computing environments:

Environment-Specific Behavior

Linux Systems: Uses /proc/meminfo for memory detection
Genepool Cluster: Automatically uses fallback memory due to resource management constraints
Limited Environments: Handles systems where memory detection fails
ulimit Compliance: Respects user-set memory limits for resource management

Error Handling

Memory Detection Failure: Falls back to default memory with warning messages
Invalid Memory Values: Uses conservative defaults to prevent out-of-memory errors
Permission Issues: Provides suggestions for resolving access problems

Legacy Status

Xcalcmem.sh is a legacy version of memory calculation functionality:

Differences from Current calcmem.sh

Simplified Logic: Uses basic memory detection without advanced cluster support
Limited Flag Support: Supports fewer Java flags compared to current version
Basic Error Handling: Simple fallback mechanisms
No Environment Detection: Lacks automatic path and environment configuration

Migration Recommendation

For new projects, use calcmem.sh instead of Xcalcmem.sh. The current version provides:

Enhanced cluster and SLURM support
More memory detection algorithms
Additional Java flag processing
More error handling and reporting

Technical Details

Implementation specifics of the memory calculation system:

Memory Formula

RAM = ((available_memory_kb - 500000) × percentage / 100) / 1024

The formula reserves 500MB for system overhead and converts from kilobytes to megabytes.

Safety Margins

System Reserve: 500MB reserved for OS and other processes
Conservative Percentage: Default 84% allocation prevents memory exhaustion
Minimum Checks: Ensures calculated memory is reasonable and positive

Performance Considerations

Heap Synchronization: Matching Xms and Xmx prevents memory allocation delays
System Integration: Respects ulimit and system constraints for stability
Execution: Bash implementation for memory calculation

Support

For questions and support:

Email: bbushnell@lbl.gov
Documentation: bbmap.org