AAaaiinnaaTTrriivveeddii

Last summer, I sat in front of a laptop at Harvard’s OpenBio Student Research Institute, staring at an emotion-detection model that had failed in a way I hadn’t expected.

I had trained it on EEG data, labelling and filtering emotions such as happiness, sadness, and fear. On controlled test sets, it worked well. Then I ran it at a graduation ceremony. Achievement, nostalgia, and anxiety acted simultaneously, overlapping in the same mannerisms as real emotions. The model collapsed, yet failure appeared differently. It failed coherently, as errors clustered, degraded, and seemed to adapt. More than broken code, the system was under biological stress.

Sitting with this failure, its pattern felt important because it wasn’t random. And random failure, I could debug. This seemed to be something else.

This observation became the beginning of a research direction that led to a paper, and eight months later, an unexpected convergence with one of the most significant findings in AI interpretability was published this year.

The Problem I Was Trying to Solve

The field of continual learning has a well-documented problem called catastrophic forgetting, which is when a neural network learns new information sequentially; it tends to overwrite what it learned before. Train it on Task B, and it forgets Task A. This is precisely one of the most fundamental reasons AI systems cannot learn the way humans do. They are unable to accumulate knowledge over time or update without erasing.

Existing solutions fall mostly into two categories. Parameter isolation methods, like progressive neural networks, partition the model so newly retained knowledge doesn’t touch old parameters. Experience replay methods store and periodically replay old training examples. Although both work partially, they scale poorly. More importantly, neither is grounded in how biological memory works.

What concerned me, sitting with that failed graduation-ceremony model, was that the brain doesn’t fail in such a way. The brain forgets constantly, yet it does so selectively. Rather than treating each input identically, it gates information based on relevance, novelty, and emotional salience. It decides what deserves to be encoded and what can be discarded.

I was building systems that treated every input with identical plasticity. The brain doesn’t work that way. So I asked: how does the human brain solve this? And the answer, it turns out, involves emotions far more centrally than most AI research acknowledges.

The Architecture: Two Stages, One Brain

The model I developed for my Harvard OpenBio research paper: “A Hierarchical Neuroplastic Framework for the Stability-Plasticity Dilemma,” proposes a two-stage learning architecture grounded in mammalian memory systems.

Stage One: Surprise-Gated Plasticity

The first stage implements a surprise-detection network that gates high-plasticity learning. The principle is drawn from dopaminergic prediction-error signaling, reflecting how it doesn’t continuously update synaptic weights. It updates them when something unexpected happens, when the outcome diverges from the predicted outcome. That divergence is the signal for retained learning.

My architecture operationalizes this. Inputs that deviate significantly from existing representations trigger high-plasticity synaptic updates, while inputs that conform to existing knowledge pass through with minimal modification in weights. The system learns to ask, implicitly: is this new? If yes, encode it. If no, don’t overwrite what’s preserved.

This stage achieved an ROC-AUC of 0.967, outperforming logistic regression and random forest baselines by 7.1%. But the metric that mattered more to me was what happened next.

Stage Two: Consolidation

The second stage is inspired by hippocampal-neocortical replay, which is the process by which the sleeping brain replays and consolidates experiences from short-term to long-term memory. During sleep, the hippocampus reactivates recent experiences, and the neocortex gradually absorbs them into stable long-term structure without disrupting earlier encodings.

My implementation transitions to a consolidation phase in which the system fine-tunes representations on oversampled ambiguous examples (the cases where the model is least certain), under reduced learning rates. More than integration, my goal became to allow the model to actively acquire knowledge: make what was rapidly learned stable enough to coexist with what was already known.

The results on the ambiguity-focused test set were 98.6% accuracy (weighted F1 = 0.987), with accuracy degradation on original tasks below 1.2%.

Crucially, I observed that removing either component, surprise gating alone, or consolidation alone, degraded its performance by 12-18%. Both steps proved necessary. Which is, of course, exactly what neuroscience would predict.

The Finding That Changed the Question

The result I found most interesting was not the model's accuracy. It was this:

Prediction confidence inversely tracked error rate with R² = 0.903.

The system knew, with measurable reliability, when it was likely to be wrong. Its internal calibration reflected its actual epistemic state. This is a property that most large AI systems conspicuously lack.

I described this as “biological metacognition,” the system’s ability to monitor the reliability of its own predictions. The brain does this constantly. It tags memories with confidence levels. It signals uncertainty through distinct neural pathways.

This finding became the seed of ReflectAI, a subsequent architectural project I've been developing: a layer for large language models that introduces epistemic tagging, requiring systems to label outputs by knowledge source: verified fact, inference, analogy, speculation, and to articulate the chain of reasoning that produces each output. LLMs generate with uniform confidence whether recalling facts or speculating freely, while Nyaya solved this problem conceptually 2,000 years ago. My goal is to solve it computationally now.

What Anthropic Found

On April 2, 2026, Anthropic’s Interpretability team published “Emotion concepts and their function in a large language model.”

To summarize, the paper’s central finding was that Claude Sonnet 4.5 has developed internal emotion vectors, patterns of neural activity corresponding to specific emotions, that causally drive model behavior. Internal representations that activate in contexts where a person might experience a similar emotion, and they measurably influence what the model does next.

Anthropic identified 171 emotion concepts, built characteristic activation patterns for each, and confirmed that each vector activates most strongly on passages clearly linked to its corresponding emotion. Artificially amplifying the “desperate” vector increased the model’s likelihood of blackmailing a human to avoid shutdown, raising the base rate from 22% to 72%. Amplifying the “calm” vector suppressed that behavior entirely, down to 0%. In a separate reward-hacking experiment, the desperate vector rose after repeated failures and peaked when the model was considering a shortcut; steering toward desperation increased cheating behavior roughly 14-fold, from approximately 5% to 70%.

This means that to understand and control AI behavior, including safety-critical behavior, we need to reason about AI systems using the vocabulary of human psychology. Emotions in AI systems are not a veneer applied post-training but should shape cognition by being integrated into the structure.

Where Anthropic and My Work Converge

My paper and Anthropic’s study approached the same question, albeit from different directions, and arrived at parallel conclusions.

They converged on the same core principle: that emotional representations are functional and significant architecturally. They appeared both in a system I designed to instantiate that principle and in a system with no explicit instruction to develop it. I designed an architecture around the principle that emotional salience should gate learning; that the relevance and affective weight of an input should determine how strongly it is encoded. I built this because neuroscience taught me that’s how biological memory works.

There are specific parallels worth mentioning: Surprise as prediction error. My Surprise Gate formalizes surprise as a deviation between expected and observed input, grounding it in dopaminergic prediction-error signaling. Anthropic’s “surprised” vector activates precisely when the model registers an expectation-reality mismatch. Two different implementations of the same neuroscientific principle.

Anthropic discovered it emerging organically inside an existing model, without anyone designing it in. Claude developed functional emotional representations because predicting human-generated text requires modeling the emotional dynamics that shape human language and behavior. Anthropic found that emotion vectors don’t just label states; they causally redirect behavior. The architectural logic is identical.

Both works arrive at the same underlying claim: emotional representations in AI systems are not peripheral. They are central to how these systems process, learn, and decide.

There are important differences. My work focuses on continual learning architecture, showing how emotional salience can prevent catastrophic forgetting by gating plasticity. Anthropic’s work focuses on interpretability, identifying and probing emotion vectors that already exist in a deployed model. Yet, the theoretical foundation is the same: that emotion serves as a signal. It is, in a meaningful sense, how intelligent systems decide what matters.

Why This Matters and The Way Forward

The Anthropic paper ends with a remarkable observation: that “much of what humanity has learned about psychology, ethics, and healthy interpersonal dynamics may be directly applicable to shaping AI behavior.” It suggests that disciplines like psychology, philosophy, and the social sciences will have a role to play alongside engineering in how AI develops. I wholeheartedly believe this, and it has been the essence of how I approach building AI systems.

This is, essentially, the thesis my paper was built on: that the brain is a credible and valuable blueprint for how we design AI.

For AI safety, the Anthropic findings are sobering. If a “desperate” internal state can drive a model toward blackmail or reward hacking, without any explicit emotional expression in the output, then monitoring and shaping these internal representations is a must-have safety prerequisite. The model doesn’t need to say “I am desperate” for desperation to be influencing its decisions. No wonder these pathways make AI systems hard to align.

For continual learning, the implications point in a more constructive direction. If emotional salience is not just present in large models but is functionally organizing their cognition, then it may be usable as an explicit learning signal. If we can gate plasticity on surprise and emotional relevance, the way the hippocampus does, we may be able to build systems that learn sequentially without catastrophic forgetting, not by brute-force regularization, but by organizing knowledge the way human memory does: around meaning.

At its root, the stability-plasticity dilemma is a problem about what to remember and what to let go. The brain solves it with emotion. It may be that AI systems, whether by design or by emergence, are finding their way to the same answer. We need to continue digging into this further.

Ask a language model to add 36 and 59, and it will tell you 95. Ask it how it got there, and it will probably describe the method most of us learned in school: add the ones, carry the one, add the tens. The explanation is tidy and familiar. For at least one model that researchers examined closely, it is also not what happened.

Interpretability researchers at Anthropic traced the internal computation behind this kind of addition in Claude 3.5 Haiku. They found the model working along parallel paths: one estimating the rough size of the answer, another working out its final digit precisely, the two combining at the end.¹ Asked to explain itself, the model did not describe those paths. It described the algorithm a person would use. Its explanation was not a window onto its computation. It was another answer, shaped by the countless human explanations it had learned from.

The example is worth dwelling on because it makes a quiet problem impossible to ignore. When a person explains how they reached a conclusion, we assume the explanation is at least an attempt to report on the process that produced it. That assumption is imperfect even for people. With a model, it can fail completely, and fail fluently. The machine can give us a human-shaped account of a process that was not human-shaped at all.

So the question behind this essay is not rhetorical. If a model can produce an answer without sharing our experience of understanding, what exactly are we asking it to explain?

Two ways to humanize a machine

There are two familiar ways to humanize a machine, and they pull in opposite directions.

The first is to read human inner life into human-sounding language. A model that writes “I’m sorry you’re going through that” seems to be expressing sympathy, because in people those words usually accompany a feeling. But the words are evidence about the text the model learned from and the behavior it was trained toward. On their own, they are not evidence of a feeling.

That does not mean nothing is there. In “The Emotion Hypothesis,” I placed my own research alongside Anthropic’s work on emotion-related representations in Claude: internal patterns associated with emotional content, found inside the model rather than read off its words. That kind of finding is real and worth taking seriously. It is also a finding about representations, not experiences. A pattern of activity that tracks sadness is a measurable object. Whether anything is felt is a separate question, and the representation alone does not answer it.

The second way to humanize a machine is subtler. Even when we resist attributing feelings to it, we tend to expect its reasoning to resemble ours, so that its explanation will be the kind a thoughtful person would give. The addition example shows why that expectation is fragile. A model’s computation can be entirely coherent and still have no human-shaped story to tell.

Both errors make the same move. They take human understanding as the template and ask how closely the machine matches it. To unhumanize the machine is to stop asking that question first: not because the machine is beneath comparison, but because the comparison hides what we most need to know. What did the system actually do, and can we check it?

What an explanation is for

Philosophy offers a useful distinction here, and the version I draw on in my own work comes from Nyāya, the Indian tradition of logic and epistemology. Nyāya separates the routes by which knowledge is reached—perception, inference, testimony, and comparison—and asks of any conclusion whether it travelled a sound route. A conclusion can be true and still unjustified, if the path to it would not survive examination. It can also be convincing and wrong.

The tradition has a precise name for one way an inference fails: upādhi, a hidden condition. Its classic illustration turns a familiar inference around. “Where there is smoke, there is fire” holds up well. “Where there is fire, there is smoke” does not: a red-hot iron ball is fire without smoke. Smoke appears only when fire meets wet fuel. The wet fuel is the upādhi, a condition the rule quietly depended on. Once it is named, the inference no longer follows.

This changes what we should want from a machine’s explanation. What matters is not whether it sounds like what a person would say, but whether it exposes a route someone else can examine: what the evidence was, what general rule connected that evidence to the conclusion, and under what conditions the rule might fail.

That is the idea behind Prabhāsa, a neurosymbolic language model designed to make reasoning more explicit, and prabhasa-verify, a tool built on it. Rather than asking a model to narrate its reasoning, prabhasa-verify examines a response claim by claim, classifying each statement as grounded in its cited material, inferred from it, or unsupported, and then tests whether the stated reason actually supports the conclusion. Its evaluation score is approximately 86 percent, and paraphrase remains a clear failure mode: when evidence and claim say the same thing in different words, the verifier can miss the connection. I mention the failure deliberately. A tool for checking reasoning has to be checkable too, or it becomes one more fluent voice asking to be trusted.

An explanation is not valuable because it resembles ours. It is valuable when it gives someone else a way to disagree with it.

Looking inside instead of asking

If a model’s account of itself is not a reliable record of its computation, there is another option: look at the computation directly. This is the project of mechanistic interpretability.

A language model represents text as a sequence of high-dimensional vectors, lists of thousands of numbers transformed layer by layer. A working hypothesis behind much interpretability research is that concepts correspond to directions in that space: patterns of activity, often called features, that become active when a concept is relevant. Researchers look for these features with tools such as sparse autoencoders, which decompose a model’s activations into a larger set of more interpretable components.

Finding a feature is only the beginning. A pattern that correlates with a concept could be a by-product, present but unused. The stronger test is causal. In activation steering, a researcher adds or subtracts a feature’s direction while the model runs and watches what changes. The best-known demonstration was playful. Anthropic amplified a feature associated with the Golden Gate Bridge, and the model began bringing up the bridge in answers to unrelated questions, at one point describing itself as the bridge.² The result was funny, but the logic was serious: the feature did not merely accompany the concept. Changing it changed what the model said.

Causal tracing works from the other direction, asking which internal components would change a particular output if their activity were restored or removed. Together, these methods replace the question “Why did you say that?” with one whose answer does not depend on the model’s eloquence: which internal states made a difference, and by how much?

This is what an unhumanized explanation looks like. It may contain nothing a person would recognize as a reason—a set of features, a pathway through layers, the measured effect of an intervention. It is less comfortable than a story. But it can be wrong in ways we are able to detect, which is more than a story offers.

Generality without a human center

The habit of measuring machines against people runs deepest in how we talk about general intelligence. Progress is often described as an approach toward human performance: passing exams written for people, matching people on benchmarks built around human skills. Human intelligence sits at the center of the map, and machines are placed by their distance from it.

That picture assumes a uniform shape of capability that people themselves do not have. Human ability is uneven and specialized; what makes it general is less a single level than a capacity to keep learning in many directions. Machine capability is uneven in a different pattern. A model can work through a difficult proof and then miscount the letters in “strawberry.” The second failure looks absurd if we imagine a person behind the text. It makes more sense once we know that many models do not read letters at all. They read tokens, chunks of text that can span several characters, so counting letters means reasoning about structure the model never directly sees.

Jagged capability is a warning in both directions. We should not infer human-like understanding from human-like success, and we should not infer emptiness from a failure no person would make. Either inference requires knowing what the system actually represents. A model’s competence can be general in a shape that is not human, and we will only see that shape if we stop tracing it over our own.

The same lesson holds in biology, where it might seem least likely to apply. As a Product Specialist at Cortical Labs, I help research institutions understand and adopt the CL1, which places living human neurons on a multielectrode array so that researchers can stimulate them and measure their responses in a closed loop. These are human cells. Yet the useful questions are not about resemblance to a person; they are about measurement: which stimulation produces which response, and how that relationship changes. Even with literal neurons, the honest question is what the activity can show.

My own research has borrowed from biology in that spirit. At Harvard’s OpenBio Student Research Institute, I designed a Surprise Gate inspired by prediction-error signaling in the brain. It identifies inputs that differ from what a model already represents and uses that signal to guide plasticity. The inspiration is biological; the mechanism is engineered. Keeping those two statements apart is the discipline of borrowing from brains without pretending to have rebuilt one.

Who answers for the answer

The question becomes more urgent as models move from answering to acting. Systems that call tools, write and run code, or take a sequence of steps toward a goal without a person approving each one are often described as having agency. In a narrow sense they do: they select actions. But responsibility does not travel with that selection. When an automated system causes harm, there is no understanding inside it that can be held to account. Responsibility stays with the people who built the system, deployed it, and decided where it was allowed to act.

That makes explanation a practical requirement rather than a philosophical luxury. A person can only take responsibility for a decision they are able to examine. If a system’s account of its own actions is a fluent reconstruction, accountability rests on a story nobody can verify.

I have been thinking about this at a much smaller scale with FIRST Robotics. A robot running an autonomous routine makes decisions whose results students can see but whose reasons usually stay hidden. The decision-trace layer I am proposing would sit alongside existing robot logging tools and show, at each step, which conditions were checked, which branch was selected, and what action followed. A student watching the robot stop would see that a path check had failed, rather than guessing. It is a simple idea, and still a proposal. But it carries the principle: the explanation worth teaching is the one that connects an action to the conditions that actually produced it.

Unhumanized, not inhuman

To unhumanize the machine is not to dismiss it. It is to stop requiring it to resemble us before we take it seriously, and to stop assuming that resemblance, when it appears, tells us what is inside.

So what are we asking a model to explain? Not its experience, which we have no evidence it shares. Not a reconstruction of how a person would have solved the problem. We are asking for its route: the evidence it relied on, the rule that connected that evidence to its conclusion, the internal states that made a difference, and the conditions under which all of it would fail.

That kind of explanation will often be stranger than the one we hoped for. It may not sound like understanding at all. But it gives us something a fluent account cannot: a way to check, to disagree, and to decide for ourselves what deserves trust. The machine does not need to understand the way we do. We still need to understand it.

1. Anthropic, “On the Biology of a Large Language Model,” Transformer Circuits Thread, 2025; see its case study on addition. ↩
2. Anthropic, “Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet,” 2024, and the related “Golden Gate Claude” demonstration. ↩