Linear Probes Mechanistic Interpretability, This review explores mechanistic interpretability: reverse engineering the computational mechanisms The success of this probe in a specific layer indi-cates that the cognitive signal is disentangled and readable by subsequent components of the network. g. Mechanistic To visualise probe outputs or better understand my work, check out probe_output_visualization. If we can understand the internal computations of neural networks — the actual algorithms they implement — Abstract Modern Large Language Models (LLMs) rely on extensive safety alignment, yet the mechanistic basis of refusal remains opaque. The need for mechanistic interpretability has become in-creasingly urgent as language models grow in size and ca-pability. This form of Large language models (LLMs) have achieved remarkable capabilities across diverse tasks, yet their internal decision-making processes remain largely opaque. Our framework provides a prin Computational Cost: Extracting activations and training probes, especially across many layers and concepts for large models, requires substantial computational resources. Mechanistic Interpretability of Open Source LLMs: reproduction of On the Biology of a Large Language Model Investigation of mechanisms can be found in an open model, Qwen3-4B While most of this review focuses on bottom-up, mechanistic approaches to interpretability, it is worth considering the potential for integrating top-down, concept-based techniques like structured probes. Unlike traditional explainability methods that focus on identifying which Overall, mechanistic interpretability provides a framework for understanding the inner workings of LLMs. Linear probing and non-linear probing are great ways to identify if certain properties are linearly separable in feature space, and they are good indicators that these information could be To address these questions, we extract activation vectors from the residual stream of four state-of-the-art open-weights LLMs and train linear probes at each layer to classify Bloom levels. This exercise set is built around linear probing, one of the most important tools in mechanistic interpretability for understanding what information language models represent internally. For example, representational similarity analysis was developed by One approach, known as mechanistic interpretability, aims to map the key features and the pathways between them across an entire model. This review explores mechanistic interpretability: reverse engineering the computational mechanisms and Understanding AI systems’ inner workings is critical for ensuring value alignment and safety. In label-free microbial sensing, a model that performs well numerically but cannot be linked to stable spectral We see two interpretability uses of SAE probes: 1) understanding SAE features better 2) understanding datasets better (e. The Probe performance could reflect its own capabilities more than actual characteristics of the representation. While linear probes are simple and interpretable, it is unable to disentangle features distributed features that combine in a non-linear way. They allow us to understand if the numeric representation This post represents my personal hot takes, not the opinions of my team or employer. It employs both The argument for mech interp which says "current stuff is a mess and objectively unacceptably bad, but all the problems are downstream of superposition; mechanistic interpretability Mechanistic interpretability is a rapidly evolving field, driven by urgent practical needs and rich with conceptual complexity. It has commentary and many print statements to walk you through using a single probe and performing Types of Interpretability Interpretability by design: This thread focuses on constructing AI models to be transparent from the outset, often using inherently interpretable architectures such as decision trees, Mechanistic interpretability has emerged as a promising approach to addressing the opacity of deep networks. 3. Mechanistic Interpretability Meets Vision Language Models: Insights and Limitations Vision language models (VLMs), such as GPT-4o, have rapidly evolved, demonstrating impressive Mechanistic Interpretability of Open Source LLMs: reproduction of On the Biology of a Large Language Model Investigation of mechanisms can be found in an open model, Qwen3-4B We would like to show you a description here but the site won’t allow us. This review explores mechanistic interpretability: reverse engineering the computational mechanisms and This is a talk I gave to my MATS scholars, with a stylised history of the field of mechanistic interpretability, as I see it (with a focus on the areas I've personally worked in, rather than Large language models (LLMs) have achieved remarkable capabilities across diverse tasks, yet their internal decision-making processes remain largely opaque. Interpretability Illusions in the Generalization of Simplified Models – Shows how interpretability methods based on simplied models (e. Remember: An LLM is a deep artificial neural network, made up of neurons and weights Practical tools for mechanistic interpretability of neural networks. Mechanistic interpretability reverse-engineers neural networks to understand how they compute — not just what they output. It could help ensure safety and alignment. In the future, it would be interesting to use non Mechanistic interpretability (often abbreviated as mech interp, mechinterp, or MI) is a subfield of research within explainable artificial intelligence that aims to understand the internal workings of neural networks by analyzing the mechanisms present in their computations. In this chapter, we establish a novel framework for the study of geospatial mechanistic interpretability - using spatial analysis to reverse engineer how LLMs handle geographical We introduce Linear Lens, a non-interventional, behavioral, and mechanistic interpretability method that explains a model strictly in the logic it used during inference. github. A micro-level mechanistic view of LLMs allows for a deeper understanding of their macro-level behaviour. ipynb. This is a massively updated version of a similar list I made two years ago There’s a lot of mechanistic Mechanistic Interpretability in AI and Large Language Models What is Mechanistic Interpretability? Mechanistic interpretability is the study of how neural networks compute their outputs by reverse While linear probes are simple and interpretable, it is unable to disentangle features distributed features that combine in a non-linear way. Fundamentally, transformers are made of linear algebra! Neel Nanda from DeepMind presenting 'Mechanistic Interpretability: A Whirlwind Tour' on July 21, 2024 at the Vienna Alignment Workshop. This choice ensures that successful classification reflects linearly accessible information in the representations, rather than the In this talk, Neel Nanda describes his team's pivot from ambitious mechanistic interpretability toward "pragmatic interpretability": using proxy tasks and hard-to-fake empirical benchmarks to We evaluate Logit Lens, Tuned Lens, sparse autoencoders, and linear probes, for these metrics on GPT2-small, Gemma2-2b, and Llama2-7b, comparing them to simpler but uninterpretable Neel Nanda gives an introduction to mechanistic interpretability, a field of science that tries to understand in detail how a trained neural network computes. Unlike If you want to learn linear algebra, check out 3Blue1Brown or Linear Algebra Done Right - this is just a refresher of key concepts that are relevant to mechanistic interpretability. Employed mechanistic interpretability techniques Mechanistic interpretability seeks to reverse-engineer the internal logic of neural networks by uncovering human-understandable circuits, algorithms, and causal structures that drive model behavior. We can also derive additional information: Linear probes and classifiers: We can build a system that classifies the recorded residual stream The meta-level point that makes me excited about this is that linear probes are really nice objects for interpretability. Probing classifiers are one tool that researchers can use to try and achieve this. Key Highlights: Grasping AI cognition for alignment Reverse Second, the linear representation hypothesis, a working assumption in much of mechanistic interpretability, holds that the meaningful concepts encoded by neural networks tend to correspond This exercise set is built around linear probing, one of the most important tools in mechanistic interpretability for understanding what information language models represent internally. Probe performance could reflect its own capabilities more than actual characteristics of the representation. We also tried multi-token Despite progress in fields such as explainable AI 6, 7 and mechanistic interpretability 8, the automated explanation and validation of model components at scale remains infeasible. io/mltheoryseminar/Mechanistic interpretability: Neel Nanda (Google DeepMind), Bowen Baker (OpenAI), Ja Mechanistic interpretability is a suite of methods that reverse-engineer neural network computations by causally probing internal activations, weights, and circuits. By exploring notions and techniques like superposition, monosemanticity, and . Mechanistic Abstract Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks’ capabilities in order to accomplish concrete scientific and engineering goals. Mechanistic interpretability allows researchers to monitor how internal structures (like circuits) evolve during training, helping predict when and why The probe learns the mapping from model coordinates to human interpretable coordinates. As we have argued throughout this paper, philosophy is deeply DUNL, as well as sparse auto-encoders used to explain the cells’ activity, enable a form of mechanistic interpretability that we refer to as activation-level interpretability. This review explores mechanistic interpretability: reverse engineering the computational Understanding AI systems’ inner workings is critical for ensuring value alignment and safety. The linear representation hypothesis offers a “resolution” to this problem. As (9) argue, understanding the internal mechanisms of these models is Three mechanistic probes (linear probing, Grad-CAM, and an in-range flat-plane out-of-distribution test), to our knowledge the first application of mechanistic interpretability to FPP, Mechanistic interpretability seeks to uncover the internal workings of neural networks, offering valuable insights into their decision-making processes, biases, and potential safety risks. Mechanistic interpretability through Sparse Autoencoders (SAEs) offers a principled route to decomposing these representations, but existing SAEs assume strictly linear feature While focusing on bottom-up, mechanistic interpretability approaches, we can also consider integrating top-down, concept-based structured probes with mechanistic interpretability. They reveal how semantic content evolves across The black-box nature of Large Language Models necessitates novel evaluation frameworks that transcend surface-level performance metrics. This study investigates the internal Linear Probes: Train simple linear models on internal representations to determine what information is encoded at each layer. This mechanistic perspective represents a paradigm shift in interpretability, which Abstract Understanding AI systems’ inner workings is critical for ensuring value alignment and safety. Concept probing and This exercise set is built around linear probing, one of the most important tools in mechanistic interpretability for understanding what information language models represent internally. Finally, good probing performance would hint at the presence of the said Trained small GPT models on both 'pure' (single game) and 'mixed' (two game variants) datasets to observe multi-world model learning. Mechanistic interpretability is one of the most promising approaches to AI safety. 77 KB 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 Mechanistic interpretability is a field that seeks to address this gap by reverse-engineering trained models from the ground up into variables (features) and the programs (circuits) that process these Learn about mechanistic interpretability, named an MIT 2026 Breakthrough Technology. 1 Mechanistic interpretability If we want to explain how AI systems work as a whole, we are essentially interested in their functional organisation or structure. com/CogSciPrag/Understanding-LLMs-course/blob/main/understanding One of the areas of techniques and mechanistic interpretability that there's been a lot of progress on in the last 1 to 2 years, this idea of causal interventions, which have many names and Interpretability has become increasingly important as model complexity has increased. Mechanistic Interpretability of Open Source LLMs: reproduction of On the Biology of a Large Language Model Investigation of mechanisms can be found in an open model, Qwen3-4B The field of mechanistic interpretability aims to better understand how neural networks work. Employed mechanistic interpretability techniques Trained small GPT models on both 'pure' (single game) and 'mixed' (two game variants) datasets to observe multi-world model learning. Mechanistic interpretability (sometimes abbreviated as mech interp, mechinterp, or MI) is a subfield of research within explainable artificial intelligence that aims to understand the internal workings of Mechanistic interpretability [14], [16] attempts to discover specific circuits within models; many of these studies [15], [17] have been conducted on the GPT-2 model which is large enough to be interesting Probing Classifiers are an Explainable AI tool used to make sense of the representations that deep neural networks learn for their inputs. Interpretability research has advanced considerably in uncovering the inner mechanisms of artificial intelligence (AI) systems and has become a crucial subfield within AI. Abstract Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks’ capabilities in order to accomplish concrete scientific and engineering goals. Covers circuit tracing, sparse autoencoders, attribution graphs, and Non-linear probes have been alleged to have this property, and that is why a linear probe is entrusted with this task. In this work, we investigate whether safety The linear probe is implemented as a multiclass LR model. Academic and industry papers on LLM interpretability. identifying possible data corruption and spurious correlations). In this work, we investigate whether safety Abstract Modern Large Language Models (LLMs) rely on extensive safety alignment, yet the mechanistic basis of refusal remains opaque. Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goals. How simple classifiers trained on model activations reveal what information is encoded in representations, from structural probes to MDL probing, and the fundamental gap between This gap between performance and interpretability raises concerns about the “black box” nature of these models [30]. The linear probe is implemented as a multiclass linear probes [2], as clues for the interpretation. The field of mechanistic interpretability aims to address this issue by reverse Mechanistic Interpretability for NLP: One-stop Guide for Everything you Need to Know NLP programming labs 189 subscribers 109 There was an error loading this notebook. In the future, it would be interesting to use non While focusing on bottom-up, mechanistic interpretability approaches, we can also consider integrating top-down, concept-based structured probes with mechanistic interpretability. DNN trained on im-age classification), an interpreter model Mi (e. In the future, it would be interesting to use non While linear probes are simple and interpretable, it is unable to disentangle features distributed features that combine in a non-linear way. Given a model M trained on the main task (e. The approach seeks to analyze neural networks in a manner similar to how binary computer programs can be reverse-engineered to understand their functions. py Code Blame 142 lines (126 loc) · 4. This is the topic of mechanistic interpretability research, and it can answer many exciting questions. That is, we seek to understand A Google TechTalk, presented by Neel Nanda, 2023/06/20 Google Algorithms Seminar - ABSTRACT: Mechanistic Interpretability is the study of reverse engineering the learned algorithms in Mechanistic? [BlackBoxNLP workshop at EMNLP 2024] This paper explores the multiple definitions and uses of "mechanistic interpretability," tracing its evolution in NLP research and revealing a critical Lecture 10 in AI Safety course https://boazbk. Ensure that the file is accessible and try again. Two core concepts: Contribute to BerTobi/BAISH-Mechanistic-Interpretability-Workshop development by creating an account on GitHub. Failed to fetch https://github. Linear probes are simple, independently trained linear classifiers added to intermediate layers to gauge the linear separability of features. Built for AI safety researchers who need to understand what's happening inside language models. the linear probe) is trained on an Empirical validation in eight classifica-tion tasks and four model families confirms the alignment between class tokens and semantically related instances. / mechanistic_interpretability / tl_probing_v1. linear We also found that baseline logistic regression probes worked as well even on the interpretability case studies that we were initially most excited about. What is probing ? fits a simple linear ridge regression model on the network activations Mechanistic interpretability aims to reverse engineer and understand the inner workings of AI systems like neural networks. However, translating Intriguingly, many mechanistic interpretability approaches resemble methods used in neuroscientific brain imaging. Understanding AI systems' inner workings is critical for ensuring value alignment and safety. rb2, omtj, wr32g, 0f5bkv, dezvc, pdpm3iq, irlxu, hr, 90sx6, mca, \