Model FLOPs Utilization (MFU)
The ratio between the observed floating-point operations per second (FLOPs) achieved during deep learning model training and the theoretical maximum peak hardware FLOPs advertised by the accelerator vendor.
MFU = Observed Throughput (TFLOPS) / Theoretical Peak Hardware TFLOPS Observed TFLOPS = (6 × Model Parameters × Tokens per Second) / Number of GPUs
Parameters & Constraints:The standard formula for calculating empirical cluster training efficiency.
Analytical Breakdown & Industry Application
Hardware vendors frequently quote theoretical peak FP16/BF16 matrix multiplication speeds (e.g. 1,979 TFLOPS for Nvidia H100 SXM5). In production distributed training clusters, actual achieved throughput is significantly lower due to memory access stalls, pipeline bubbles, and inter-node all-reduce collective communications.
MFU provides an unvarnished metric of cluster efficiency: an MFU of 45%–52% represents world-class distributed optimization (e.g. Megatron-LM), while naive distributed scripts frequently operate at sub-30% MFU, effectively wasting half the capital expenditure invested in GPU hardware.
Critical Takeaways for Practitioners
- •Accounts for tensor-parallel and pipeline-parallel communication overhead.
- •An MFU above 45% in a multi-thousand GPU cluster indicates excellent InfiniBand/RoCE network tuning.
- •Directly determines the dollar cost per million tokens trained on enterprise clusters.
Primary Technical & Regulatory Sources
Foundational research defining MFU benchmarking and activation checkpointing formulas.
Architectural benchmark data on communication-to-computation overlap in ZeRO stages.

