MemDetect

Basic Usage

memdetect.sh [memory_specification] [percentage] [mode]

Memory detection utility that can be run directly or sourced by other scripts. When run directly, outputs the detected memory in megabytes.

Function Interface

The primary function detectMemory() accepts three optional arguments:

Function Arguments

$1 - memory_specification: Default memory allocation with unit suffix (e.g., "3200m", "4g", "500k"). If not provided, uses 3200MB default.
$2 - percentage: Percentage of available memory to use (integer, e.g., 75). Default is 45% for shared nodes, 85% for exclusive nodes.
$3 - mode: Memory allocation mode: "auto" (default), "partial", or "fixed". Fixed mode returns the specified memory without detection.

Memory Detection Configuration

The script uses several internal constants and environment variables for memory detection:

Internal Constants

DEFAULT_MEM_MB=3200: Default memory allocation in megabytes when detection fails or for fallback scenarios.
DEFAULT_MEM_PERCENT_SHARED=45: Default percentage of memory to use on shared compute nodes (conservative allocation).
DEFAULT_MEM_PERCENT_EXCLUSIVE=85: Default percentage of memory to use on exclusive compute nodes (aggressive allocation).
RESERVED_MEM_KB=500000: Reserved memory in kilobytes (500MB) for operating system and other processes.

Environment Variables

RQCMEM: User-specified memory override in megabytes. If set and greater than 0, bypasses automatic detection.
SLURM_MEM_PER_NODE: SLURM scheduler memory allocation per node in megabytes. Used to constrain memory detection on SLURM systems.
OSTYPE: Operating system type. Used to determine platform-specific memory detection method (Darwin for macOS).

Memory Detection Modes

Detection Modes

auto: Automatic detection using platform-specific methods. Applies full percentage to available memory after reserving system memory.
partial: Similar to auto mode but uses more conservative memory allocation. Currently uses same percentage as auto mode.
fixed: Returns the specified memory amount without any system detection. Useful for testing or when exact memory control is needed.

Platform-Specific Detection Methods

Linux Memory Detection

MemAvailable: Primary method using /proc/meminfo MemAvailable field (most accurate, available in newer kernels 3.14+).
Free+Buffers+Cached: Fallback method for older kernels: adds MemFree + Buffers + Cached from /proc/meminfo.
Virtual Memory Check: Safety check using CommitLimit - Committed_AS to ensure virtual memory constraints are respected.

macOS Memory Detection

Total Memory (sysctl): Gets total physical memory using sysctl hw.memsize, then estimates 65% as available (conservative).
Free Memory (vm_stat): More accurate method using vm_stat to get free and inactive pages, converted to kilobytes using page size.
Conservative Approach: Uses the more conservative estimate between total memory percentage and vm_stat free memory.

System Constraints and Limits

Constraint Handling

ulimit -v: Respects user's virtual memory limit. If set and not "unlimited", constrains memory detection to this value.
SLURM Integration: Automatically detects SLURM_MEM_PER_NODE and constrains memory allocation to scheduler-assigned limits.
Minimum Memory Check: Ensures detected memory is at least 100MB; falls back to default if detection yields very low values.
Reserved Memory: Always reserves 500MB for operating system and other processes before calculating allocation.

Parameters

MemDetect accepts up to three positional parameters when called directly:

Command Line Parameters

memory_specification: Default memory allocation with unit suffix (e.g., "3200m", "4g", "500k"). If not provided, uses 3200MB default.
percentage: Percentage of available memory to use (integer, e.g., 75). Default is 45% for shared nodes, 85% for exclusive nodes.
mode: Memory allocation mode: "auto" (default), "partial", or "fixed". Fixed mode returns the specified memory without detection.

Examples

Direct Execution

# Run memory detection with defaults
memdetect.sh
# Output: Detected memory: 6400MB

# Specify custom default and percentage
memdetect.sh 4g 60
# Uses 4GB default, 60% of available memory

# Use fixed mode
memdetect.sh 8g 0 fixed
# Returns exactly 8192MB regardless of system memory

Direct execution returns detected memory amount to stdout for use in other scripts.

Sourcing in Scripts

# Source the script to access detectMemory function
source memdetect.sh

# Call the function with parameters
detectMemory "6g" 70 "auto"
echo "Allocated memory: ${RAM}MB"

# Use environment variable override
export RQCMEM=4096
detectMemory
echo "Memory: ${RAM}MB"  # Will output 4096MB

When sourced, provides the detectMemory function that sets the RAM variable.

Integration with BBTools

# Typical usage in BBTools scripts
source memdetect.sh
detectMemory "3200m" 75 "auto"
java -Xmx${RAM}m -cp $CLASSPATH jgi.SomeClass $@

Standard pattern for BBTools scripts to automatically determine optimal Java heap size.

Algorithm Details

Memory Detection Strategy

The memory detection algorithm follows a hierarchical approach to determine the optimal memory allocation:

1. Override Checking

Fixed Mode: Returns specified memory without any system queries
RQCMEM Variable: Uses manually specified memory if set and positive

2. System Constraint Discovery

ulimit Detection: Queries user's virtual memory limit using ulimit -v
SLURM Integration: Reads SLURM_MEM_PER_NODE for scheduler constraints

3. Platform-Specific Detection

Linux: Prefers MemAvailable field from /proc/meminfo (kernel 3.14+), falls back to Free+Buffers+Cached calculation for older systems
macOS: Uses sysctl hw.memsize for total memory, vm_stat for accurate free memory calculation
Fallback: Uses default values when platform detection fails

4. Constraint Application

Applies the most restrictive constraint from SLURM allocation, ulimit setting, or detected system memory.

5. Memory Calculation

Final allocation = ((available_memory - reserved_memory) × percentage) ÷ 1024 MB

Always reserves 500MB for system processes
Applies user-specified or default percentage
Converts from kilobytes to megabytes for Java heap sizing

6. Safety Validation

Minimum Check: Ensures result is at least 100MB
Fallback Protection: Uses default values if calculated memory is too low
Silent Operation: Provides warnings only when not in silent mode

Virtual Memory Considerations

On Linux systems, the script also checks virtual memory availability using CommitLimit and Committed_AS from /proc/meminfo. This prevents over-allocation in systems with memory overcommit restrictions and provides a more conservative estimate when virtual memory is constrained.

macOS Memory Estimation

macOS memory detection is inherently more complex due to the system's aggressive memory management. The script uses a dual approach:

Conservative Estimate: 65% of total physical memory
Detailed Analysis: vm_stat parsing to calculate free + inactive pages
Final Decision: Uses the more conservative of the two estimates

HPC Environment Integration

The script automatically integrates with HPC schedulers:

SLURM: Reads SLURM_MEM_PER_NODE to respect job allocation limits
Future Extensible: Architecture allows easy addition of other schedulers (PBS, LSF, etc.)
Constraint Enforcement: Never exceeds scheduler-allocated memory

Performance Characteristics

Execution Speed

Fast Execution: Typically completes in under 50 milliseconds
Minimal System Calls: Uses direct /proc/meminfo parsing on Linux
Caching Friendly: Results can be cached by calling scripts

Memory Usage

Lightweight: Uses minimal memory (under 1MB) for execution
No Dependencies: Pure bash with standard Unix utilities
Environment Safe: Does not modify global environment

Accuracy Considerations

Linux MemAvailable: Highly accurate for systems with kernel 3.14+
Conservative macOS: Intentionally conservative to prevent OOM conditions
Scheduler Aware: Respects system resource management policies

Integration with BBTools

Standard Integration Pattern

Most BBTools scripts follow this pattern for memory detection:

# Source memory detection
source "$(dirname "$0")/memdetect.sh"

# Detect memory based on tool requirements
detectMemory "$defaultmem" "$mempercent" "$memmode"

# Use detected memory for Java heap
java -Xmx${RAM}m -cp "$CLASSPATH" $MAINCLASS "$@"

Tool-Specific Customization

Memory-Intensive Tools: Use higher default allocations (e.g., BBMap uses 85%)
Lightweight Tools: Use lower defaults or fixed small allocations
Streaming Tools: May use minimal memory regardless of system capacity

Silent Mode Operation

The script respects the silent variable set by javasetup.sh to suppress informational messages during automated execution.

Troubleshooting

Common Issues

Low Memory Detection: If detection yields very low memory, check system memory usage and consider using fixed mode with a reasonable allocation.
SLURM Conflicts: On SLURM systems, ensure your job allocation matches expected memory requirements. The script will constrain to SLURM limits.
macOS Variations: macOS memory detection may vary significantly based on system load. Results are intentionally conservative.
Older Linux Systems: Systems with kernels older than 3.14 will use the fallback Free+Buffers+Cached calculation, which may be less accurate.

Debugging

Manual Override: Use RQCMEM environment variable to override automatic detection: export RQCMEM=4096
Fixed Mode Testing: Use fixed mode to test with specific memory allocations: memdetect.sh 8g 0 fixed
Verbose Output: Set silent=0 in environment to see detection process messages (uncomment debug lines in source for full verbosity)

Support

For questions and support:

Email: bbushnell@lbl.gov
Documentation: bbmap.org