Oberseminar Mathematics of Artificial Intelligence

How attention learns structure from data?Oberseminar Mathematics of Artificial Intelligence

by Prof. Claire Boyer (Université Paris-Saclay)

→ Europe/Berlin
Endenicher Allee 60/1-016 - Lipschitzsaal (Mathezentrum)

Endenicher Allee 60/1-016 - Lipschitzsaal

Mathezentrum

90
Description

Transformer architectures have demonstrated remarkable empirical success, yet their ability to extract statistical structure from data remains only partially understood. In this talk, I will present recent work shedding light on attention mechanisms through a statistical and operator-theoretic lens.
Based on large-prompt asymptotics, one can show that softmax attention converges to a linear operator acting on the input-token distribution under Gaussian assumptions. This regime enables a precise analysis of both the outputs and the training dynamics using concentration arguments, revealing a surprising bridge between nonlinear softmax attention and tractable linear models.
These results suggest that attention can be understood as a flexible statistical operator that adapts to the underlying data distribution, providing a unifying framework to study its role in representation learning and in-context inference. I will conclude by discussing the connections between transformer architectures and principal component analysis, using this setting to illustrate the power of concentration methods in high-dimensional statistics.

Organized by

Alexander Effland, Illia Karabash, and Floris van Doorn, jointly with Anton Bovier