Cotangent: Preserving Reduction Arithmetic in Batched Gradient Handling
Abstract
Gradient handling can occupy a substantial fraction of small-model training on an accelerator even when the optimizer is already fused. Batching this work is straightforward in real arithmetic but can change floating-point reductions. We study a gradient layout that preserves the native two-level clipping norm while batching equal-length tensors on PyTorch's Metal backend. The method retains every model derivative and applies native fused AdamW to consistently packed parameter and gradient coordinates. An independently frozen experiment with 100 paired seeds reduces time to a fixed validation target by 15.03% (99.5% confidence interval, 13.50–16.56%) relative to eager native fused AdamW, while meeting a predeclared 0.01-bit-per-byte quality tolerance. A separately frozen 128-run extension isolates norm batching from packing and evaluates four configurations against training-only-selected compiled baselines. Using the same compiled model, mean training-time reductions range from 23.55% to 34.07%. The extension's descriptive quality intervals do not establish the 0.01-BPB tolerance for every configuration. Numerical checks cover complete model updates, optimizer states, reduction boundaries and composition with the compiler. The results are specific to one Apple M5 Pro, the pinned PyTorch backend and byte language models; the study does not establish performance on other accelerators or model scales.
Citation
Vahedian, Atila (2026). Cotangent: Preserving Reduction Arithmetic in Batched Gradient Handling. Technical report, version 0.6.0.
@misc{vahedian2026cotangent,
author = {Atila Vahedian},
title = {Cotangent: Preserving Reduction Arithmetic in Batched Gradient Handling},
year = {2026},
note = {Technical report, version 0.6.0},
url = {https://atilavahedian.github.io/cotangent/paper.html}
}