Abstract: While foundation models have gained considerable attention in core AI fields such as natural language processing (NLP) and computer vision (CV), their application to learning complex responses of physical systems from experimental measurements remains underexplored. In physical systems, learning problems are often characterized as discovering operators that map between function spaces, using only a few samples of corresponding function pairs. For instance, in the automated discovery of heterogeneous material models, the foundation model must be capable of identifying the mapping between applied loading fields and the resulting displacement fields, while also inferring the underlying microstructure that governs this mapping. While the former task can be seen as a PDE forward problem, the later task frequently constitutes a severely ill-posed PDE inverse problem.

In this talk, we will explore the development of a foundation model for physical systems, by learning neural operators for both forward and inverse PDE problems. Specifically, we show that the attention mechanism is mathematically equivalent to a double integral operator, enabling nonlocal interactions among spatial tokens through a data-dependent kernel that characterizes the inverse mapping from data to the hidden PDE parameter field of the underlying operator. Consequently, the attention mechanism captures global prior information from training data generated by multiple systems and suggests an exploratory space in the form of a nonlinear kernel map. Based on this theoretical analysis, we introduce a novel neural operator architecture, the Nonlocal Attention Operator (NAO). By leveraging the attention mechanism, NAO can address ill-posedness and rank deficiency in inverse PDE problems by encoding regularization and enhancing generalizability. To demonstrate the applicability of NAO to material modeling problems, we apply it to the development of a foundation constitutive law across multiple materials, showcasing its generalizability to unseen data resolutions and system states. Our work not only suggests a novel neural operator architecture for learning an interpretable foundation model of physical systems, but also offers a new perspective towards understanding the attention mechanism.