⚓ ANCOR: AI_🤖 Neuro_🧠 CogSci_⚙ Research talks

Post-training alignment techniques are now standard parts of LLM development and are presumed to contribute to the human-like abilities of models. In this talk, I will present two studies examining whether LLMs' response distributions capture the diversity and value pluralism important for their deployment in human-centered applications. In the first, we evaluate claims that models can replace humans in behavioral research using a novel metric for conceptual diversity in synthetic LLM "populations" that relates individual-level variability to population-level variability. We use this approach to evaluate non-aligned and aligned LLMs on two domains with rich human behavioral data and find that aligned models display less conceptual diversity than their instruction fine-tuned counterparts. In the second work, we use a leading cognitive model of polite speech to systematically evaluate value trade-offs in two model settings: degrees of reasoning budget in frontier black-box models, and RL post-training dynamics of open-source models. We find interpretable patterns of informational, social, and presentational utilities emerging from low-level training decisions, and that these patterns shift in predictable ways when models are prompted to prioritize certain goals. Finally, I'll share thoughts on future directions for these works, including improving LLM personalization, generating hypotheses about other social behaviors like sycophancy, and guiding value trade-offs during model development.

None

None

TBD

Animals exhibit behavioural flexibility by making reward-optimal decisions in unfamiliar situations. Computing value from minimal reward feedback requires leveraging known structure in the world. The Prefrontal Cortex (PFC) plays a pivotal role in both value computation and learning world structure, however mechanistic explanations for each---within the frameworks of Reinforcement Learning and Cognitive Maps---are largely distinct. We develop a mechanistic model of PFC that unifies value and structure through behavioural flexibility. In this model, value acts as a control signal that gates structured working memory representations of the current context in order to ultimately control behaviour. This is a significant departure from the conventional perspective of value as an action readout. Critically, this model generalises to novel contexts without need for additional synaptic plasticity. We demonstrate that recurrent neural networks (RNNs) meta-trained with RL recapitulate this exact model on various value-based tasks. These results not only provide a parsimonious explanation of both the value and schema frameworks of PFC function, but also a mechanistic understanding of PFCs role in flexible decision making.

Many of the robustness properties that are required for real-world applications of AI would be realized by a model that has understood the world. But it is unclear how to measure -- let alone define -- understanding. This talk will propose theoretically-grounded definitions and metrics that test for a model's implicit understanding, or its world model. We will focus on two kinds of settings: one where implicit world models are tested behaviorally, and another that tests a model's representation. These exercises -- in applications ranging from LLMs learning the rules of games to testing whether models acquire Newtonian mechanics -- demonstrate that models can make highly accurate predictions with incoherent world models, revealing their fragility.

Representation alignment is often seen as another optimization objective in the traditional pipeline of deep learning. You create an objective function that encodes your alignment metric and then train or fine-tune a network end-to-end. In this talk, I argue that representations can be steered in alternative ways, allowing us to break this cycle. These methods range from aligning models across different modalities without any training to observing significant benefits when aligning to an untrained model. I'll begin by showing that cross-modal alignment is achievable during inference alone, without any training. I’ll also show that progressive alignment enables capabilities remain intact as we transfer between very different architectures. In short, there are more ways to align than initially thought, so you should align things.

What is a word to an LLM? Individual tokens are often semantically unrelated to the meanings of the words/concepts they comprise, meaning that LLMs must build up representations of word meaning in-context that are separate from token identities. In this talk, I show how these multi-token words can be used as a wedge to separate conceptual representations from literal token representations. I describe our "dual-route theory of induction", which argues that models copy literal token information in parallel with "fuzzy" word representations, and show that this "fuzzy" concept induction is vital for other tasks, like translation. Finally, I describe our recent work analyzing the geometry of these independent subspaces, which suggests that they are likely useful for more than just copying.

How does the visual system develop category-selective regions for faces, bodies, scenes, and words? And how can we tell whether our deep neural network (DNN) models actually capture the feature tuning that arises in these brain areas? In this talk, I will first show that contrastive learning over large-scale natural image sets naturally gives rise to category-selective units for faces, scenes, bodies, and words, even in the absence of any category-specific learning rules or inductive biases. These emergent selective units have dissociable functional roles in object recognition when lesioned, and can predict responses in corresponding selective areas of human ventral visual cortex. These findings support a unifying account of category representation that bridges longstanding debates between modular and distributed theories of high-level vision. Building on this framework, I will then introduce 'parametric neural control' as a novel, more stringent test of DNN-brain alignment. Many DNN encoding models may show near-equal performance in predicting visual responses, while relying on fundamentally different features and computations. We demonstrate this using an interpretability technique called feature accentuation (Hamblin, Fel, et al., 2024), which enables us to synthesize stimulus sets that systematically vary along model-specific encoding axes, and, to then test each model's ability to precisely modulate neural responses in macaque inferotemporal cortex. Strikingly, we find that DNNs with equivalent encoding scores on natural images can show marked differences in their capacity to control neural responses using these targeted image manipulations. This approach therefore provides a new means to arbitrate between models, requiring a stronger commitment to feature tuning properties in local parts of the natural image manifold. Overall, these studies provide an updated deep learning paradigm for understanding the emergence and function of high-level visual representations in greater detail.

Recent experimental breakthroughs have paved the way for collecting “big” neural datasets through the simultaneous recording of the activity in thousands of neurons. However, our understanding of the fundamental principles governing neural activity at the population level remains sparse and requires the establishment of appropriate theoretical frameworks. In particular, understanding how neural systems process information through high-dimensional representations presents a fundamental open challenge. A parallel issue emerges in the context of artificial neural networks, that operate efficiently via the interaction of billions of artificial neurons. A commonly adopted approach involves the analysis of statistical and geometrical attributes of neural representations as population-level mechanistic descriptors of task implementation. One of these population-geometry metrics is the invariant-object classification capacity. However, this metric has been so far limited to linearly separable settings. In the first part of the talk, I will present a theoretical framework that overcomes this limitation leveraging contextual gating of the input. (Reference: F. Mignacco, C.-N. Chou, and S. Chung. Nonlinear classification of neural manifolds with contextual information. Physical Review E 111.3 (2025): 035302). Training machine learning models relies on various optimization strategies to enhance performance. These include optimization algorithms with adaptive hyper-parameters, time-dependent selection of training examples, and model refinement through dynamic architectures. While these strategies aim to accelerate training and steer models toward solutions with good generalization properties, they often rely on trial-and-error heuristics and lack a solid theoretical foundation. Furthermore, machine learning problems are inherently high-dimensional—in terms of dataset size, input dimensions, and model parameters—challenging meta-optimization techniques that can suffer from the curse of dimensionality. Recent advances in the statistical physics of neural networks have provided powerful tools to capture high-dimensional training dynamics through low-dimensional effective equations that track the evolution of key order parameters. In the second part of the talk, I will present how to integrate dimensionality-reduction techniques from statistical physics with control-theoretic methods to identify optimal training strategies, focusing on continual learning. (Reference: F. Mori, S. Sarao Mannelli, F. Mignacco. Optimal protocols for continual learning via statistical physics and control theory. arXiv preprint arXiv:2409.18061 (2024). Accepted at ICLR 2025).

TBD

The success of Classical (i.e., symbolic) linguistic theories suggests that at least in the linguistic domain, the mind constructs structured symbolic representations and processes them with rules of symbol manipulation. However, modern Connectionist (i.e., neural) models may provide an alternative foundation for linguistic computation: Connectionist models often lack built-in mechanisms for Classical representation and processing, yet they perform impressively on a wide array of linguistic tasks. Connectionist models may not challenge the Classical approach if they implement Classical computers. This is no mere theoretical possibility: when trained on simple symbol manipulation tasks, small Connectionist models develop structured symbolic representations, a key aspect of Classical computation. This raises the possibility that Connectionist models trained on natural data also develop such representations. However, structured representation is not sufficient for Classical computation. The processing of the structured representations by the Connectionist models must involve abstract symbol manipulation of the sort that Classical theories posit. Else, Connectionist models may still challenge the Classical account of human linguistic capacities. We study this question by testing if Connectionist models trained on simple symbol processing tasks develop Classical processing mechanisms. We find evidence suggesting that Connectionist models that succeed on such tasks do implement Classical models. To the extent that such findings generalize to models trained on naturalistic data, such a result would suggest that modern Connectionist models do not challenge Classical theories of human language.

Neural networks' scaling has been argued to yield sudden learning of capabilities (a.k.a. emergent abilities). In this talk, I will first summarize our recent work on formal models that help explain the mechanisms underlying such sudden learning via data scaling, implicating the compositional nature of a task and formation of structured representations that are shared across several tasks involved in the broader data composition. Then, focusing on in-context learning (ICL)---one such suddenly learned capability---I will demonstrate the precise configurations used for training can lead to learning of fundamentally different algorithms for performing an ICL task. This indicates the phenomenology of ICL established in past work may not be universal. Further, I will discuss how merely scaling the context size can lead to a crossover between different ICL algorithms used by the model. This can be explained via a competition of algorithms lens, which also yields a new theory on the transient nature of ICL.

Recurrent neural networks can generate dynamics, but in sensory cortex it has been unclear if any dynamic processing is supported by the dense recurrent excitatory-excitatory network. Here we show a new role for recurrent connections in mouse visual cortex: they support powerful dynamical computations, but by filtering sequences of input instead of generating sequences. Using two-photon optogenetics, we measure neural responses to natural images and play them back, finding inputs are amplified when played back during the correct movie dynamic context— when the preceding sequence corresponds to natural vision. This sequence selectivity depends on a network mechanism: earlier input patterns produce responses in other local neurons, which interact with later input patterns. We confirm this mechanism by designing sequences of inputs that are amplified or suppressed by the network. These data suggest recurrent cortical connections perform predictive processing, encoding the statistics of the natural world in input-output transformations.

Predicting upcoming events is critical to our ability to effectively interact with ourenvironment and conspecifics. In natural language processing, transformer models, which are trained on next-word prediction, appear to construct a general-purposerepresentation of language that can support diverse downstream tasks. However, westill lack an understanding of how a predictive objective shapes such representations. Inspired by recent work in vision neuroscience Hénaff et al.(2019), here we test ahypothesis about predictive representations of autoregressive transformer models. In particular, we test whether the neural trajectory of a sequence of words in asentence becomes progressively more straight as it passes through the layers of thenetwork. The key insight behind this hypothesis is that straighter trajectories shouldfacilitate prediction via linear extrapolation. We quantify straightness using a 1-dimensional curvature metric, and present four findings in support of the trajectorystraightening hypothesis: i) In trained models, the curvature progressively decreasesfrom the first to the middle layers of the network. ii) Models that perform better onthe next-word prediction objective, including larger models and models trained onlarger datasets, exhibit greater decreases in curvature, suggesting that this improvedability to straighten sentence neural trajectories may be the underlying driver ofbetter language modeling performance. iii) Given the same linguistic context, thesequences that are generated by the model have lower curvature than the groundtruth (the actual continuations observed in a language corpus), suggesting thatthe model favors straighter trajectories for making predictions. iv) A consistentrelationship holds between the average curvature and the average surprisal ofsentences in the middle layers of models, such that sentences with straighter neuraltrajectories also have lower surprisal. Importantly, untrained models don’t exhibitthese behaviors. In tandem, these results support the trajectory straighteninghypothesis and provide a possible mechanism for how the geometry of the internalrepresentations of autoregressive models supports next word prediction.