Nelson's Modded-NanoGPT fork runs beyond H100 for class demos

The fork, published by Mark J. Nelson in May 2026, ports modded-nanogpt off its Hopper-only assumption so the training code also runs on Ampere and Ada Lovelace Nvidia cards. The stated purpose is classroom demos.

"Upstream assumed Hopper-class GPUs, specifically H100; this fork also runs on Ampere and Ada Lovelace architectures," Nelson writes in the README. The hardware-specific pieces are replaced with "a mix of pure PyTorch fallbacks and BF16 Triton kernels in place of kernels that assume FP8, TMA, and/or sm90 inline assembly," alongside a PyTorch flex_attention path with sliding-window block masking when Flash Attention 3 is unavailable. Validation batch size is now picked from available vRAM, since the upstream value "was only suitable for 80GB+ cards."

For the teaching context, the fork periodically generates sample text during a run so students can watch generation quality improve, and inserts quick intermediate validation passes for a quantitative read. Nelson says he has stopped tracking the upstream repo because "upstream has doubled down even more on speedrunning *specifically* the setting of 8xH100 GPUs."

Only `train_gpt.py` has been ported. `train_gpt_medium.py` is unchanged.