Activation Alignment Paper Shrinks TabPFN-3 and TabFM Context With a Linear Aligner Trained in Minutes

A new paper proposes activation alignment, a lightweight linear transformation trained on synthetic unlabeled data that maps a data-constrained student's activations toward those of a full-context teacher — closing much of the inference-time cost gap in tabular in-context learning. Evaluated on 38 TabArena classification datasets with TabPFN-3 and TabFM, the aligner trains in seconds to minutes on commodity hardware (no GPU required) and recovers nearly half the teacher's low-data advantage across all context budgets.