How the brain uses memory to imagine what might have been

A month ago philosopher Felipe De Brigard received a strange phone call. “Are you in a private place where you can have a private conversation?” the voice on the other end asked.

He was on a work retreat surrounded by colleagues. “Just say you need to talk about your taxes,” the caller said. De Brigard excused himself to find a private space, panicking the whole time about his taxes—but it was actually the MacArthur Foundation on the line. De Brigard had been awarded a prestigious “genius grant,” an $800,000 fellowship with no strings attached.

“I just was floored. And I had to put everything back together and walk back in like nothing happened,” says De Brigard, who is a professor at Duke University and originally from Bogotá, Colombia. He is only the second philosopher to be awarded the fellowship in the past 30 years.

On supporting science journalism

If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

De Brigard would be the first to tell you that his recollection of that day may be flawed. As a philosopher and cognitive scientist, he studies the fallibility of memory. Our brains don’t work like digital cameras recording our experiences to a mental hard drive, he explains. We store only some parts of our experience—gestures, stories, the gist. When we go to retrieve those memories, our brains need to fill in the gaps.

De Brigard’s research shows just how intertwined our brain’s memory is with imagination. We use some of the same mental machinery to reconstruct the past as we do to imagine the future—and to imagine what might have been.

Scientific American spoke with De Brigard about why false memories form, why we dwell on the roads not taken, how forgiveness influences memory and whether we should worry about artificial intelligence becoming conscious.

[An edited transcript of the conversation follows.]

What big question about memory first intrigued you?

I had the huge fortune of being in grad school at a moment in which there were some massive discoveries in episodic memory, or our capacity to remember specific experiences from our past. In 2007 there were three papers that sort of solidified a hunch that the renowned psychologist Endel Tulving had in the 1980s, which was that our capacity to remember the past is also needed for thinking about the future. Tulving had a very famous case report with a patient known as K.C. with hippocampal amnesia. When you asked him to remember an event that happened in his life, he drew a blank. But if you asked him to think about what he was going to do, say, next Christmas, he also drew a blank. It’s not that his imagination was broken across the board—he could imagine lots of things—but things that were part of his personal trajectory, both from the past and the possible future, seemed to be impaired.

So these three papers basically solidified this idea that the hippocampus and the network that we use for episodic recollection are deployed when we think about possible futures. Now, my question was, this is a very asymmetric way of thinking about personal time. When you remember the past, that’s what actually happened. But when you imagine the future, you’re imagining lots of possibilities, right? I started thinking, what if the same mechanisms are required to think about not only what actually happened but also what could have happened in the past? I call that episodic “counterfactual” thinking because it’s contrary to fact. And sure enough, we did a bunch of experiments, and we realized that the hippocampus is required for imaginations about possibilities that pertain to oneself.

Why would memory and imagination would be entwined? Why would we evolve that way?

So, two things: The first one is that you cannot store absolutely every detail of every experience. This suggests your brain needs to employ some computational shortcuts when it’s trying to remember the past. By filling in those gaps at the time of retrieval with the most likely possibility, your memory system can actually be quicker and be more flexible and work online in a timely manner.

The second reason is that you want your memory system and your imagination of possibilities to be highly connected. To use an expression that the philosopher Karl Popper used, we generate hypotheses so that they can die in our stead. We want to imagine alternative ways in which past events could have happened or might happen in the future to hedge future uncertainty. And if we have them really closely connected to our memory mechanisms, then we can store those mental simulations.

There are so many pieces of fiction that exercise this kind of counterfactual thinking, exploring what might have happened—like the alternate history of The Man in the High Castle or even in rom-coms such as Sliding Doors. What do you make of our tendency to dwell on these stories?

I think one of the reasons why fiction is so important in life is precisely because it allows you to inhabit those possibilities and learn from them. Sometimes I give this talk that that I call “The Moral Demands of Imagination.” I think that we are required to employ imagination in all sorts of different tasks in life, and sometimes there are moral demands to imagine things properly. Suppose that you are in jury duty, and they ask you to imagine what the defendant could have done in a particular situation. Suppose that the defendant is like me, a middle-aged Latino who moved to the U.S. in his 20s, and he did something wrong. If they asked me as a juror to imagine what this person could have done instead of what they did, I’m going to have a ton of experiences that I can draw from to generate that mental simulation. By contrast, if someone asks me, “Can you imagine what this Iranian woman in the 1920s would have done?” I have no idea what it would be like to be her. Absent the actual experience, the next best thing is to read about it.

We’ve been talking about the brain a lot like a computer—it runs computations and makes inferences based on probabilities, like large language models do. Is that a superficial similarity to you, or is there a deeper likeness between our brains and AI?

It is actually a really great question because it’s something that I myself have struggled with. Twenty years ago, when I first started thinking about the mind as sort of like the “software of the brain,” I was convinced. I was absolutely convinced that neurons really computed, that they were like biological microchips and that you could sort of reverse engineer the mind like you could a computer.

Someone who has influenced me quite a bit here is a philosopher called Mazviita Chirimuuta. She wrote an extraordinarily beautiful book called The Brain Abstracted. It made me think about how many idealizations [such as the brain as a computer] are super helpful, even if they’re wrong. I think computational models to explain, among other things, memory retrieval are extremely helpful. Whether the model is actually capturing what the neurons do, I don’t know, and I think it actually probably doesn’t. I don’t think that neurons are the only thing that matters for cognition. And I think the jury is still out as to whether neurons are actually computing or if it is best to think of neurons as computers. But many of the computational models that we have are extremely useful—in fact, they are part of the inspiration that we have for AI systems.

You’ve written a lot about consciousness, including criticizing a leading theory as pseudoscience. What do you make of the claims about AI consciousness?

This is interesting. Last semester I taught a class on consciousness and AI, and I didn’t have a view one way or another. I am a little bit more worried about whether AI could be agential, as in agency. In almost everything that we see in the world, agency and consciousness are highly correlated. We tend to think of them together, but I think that they are separable. And it has become pretty clear that there are many AI systems that seem to exhibit something like agency, and that is going to require a set of regulations and rules and ways of interacting with them that are going to be challenging. Whether or not they’re conscious is a little bit irrelevant to the decisions that we have to make right away regarding their agency. I think that that’s the more urgent question.

That makes sense—you don’t need to get bogged down in most the “hard problem” of consciousness and phenomenology to address the urgent questions involving AI right now.

Exactly. I don’t know how apt this comparison is, but I grew up with bullfighting in Colombia, and it seemed like an extraordinarily violent spectacle. Of course the bulls were suffering. But imagine we discover that in order to experience pain you need to have a particular pain receptor, and for whatever reason, the bulls are the only animals on Earth that don’t have it. The whole thing wouldn’t be less monstrous.

I think that it is very hard to separate consciousness and subjectivity from agency, and I think that a similar thing would be probably an exercise that we have to do to understand what to do with AIs that are becoming agents.

Speaking of Colombia, you are currently working on a research project, following the peace agreement reached 10 years ago, on memory and forgiveness with victims of political violence there. Is that what you’ll use the MacArthur Fellowship funds for?

Yes, I think that the majority of the award is going to go toward the memory and forgiveness project in Colombia. So far, the evidence seems to support what we call the emotional fading account: namely that forgiving helps us to modify the [emotional] reactions that we have toward the memories of wrongdoings, without necessarily making us forget the details of the event.

This is very important, especially in the context of postconflict Colombia, because we don’t want to forget what happened. And part of what we are showing is that forgiveness does not involve the forgetting of the memory but the emotional reappraisal of the original memory. Our goal with this award would be to continue that research and to try to explore now strategies in which we can employ emotional reappraisal techniques to bring people closer to forgive and reconcile. I am also profoundly interested in the hippocampus in the brain, and for me it’s still a big puzzle to try to understand exactly how this structure in the brain brings together both space and time. So I have a number of hypotheses and experiments that I want to conduct to try to fully understand the functioning of the hippocampus.

This might seem like an obvious question, but why is forgiveness a good?

I don’t think forgiveness is good all the time. I think that I think that forgiveness is a really efficient strategy to regulate your emotions and your reactive attitudes toward the perpetrators of past wrongdoings. In the case of Colombia, and in particular in the regions of Montes de María, which is where I work, because of the peace process, many of the perpetrators are coming back to live in the same communities with the same victims that they once victimized. It’s very important that we explore ways in which the emotions can be sort of mollified. Forgiveness is an extremely good way of doing that, but it’s not the only way, nor is it mandatory. That also respects the wishes of the participants that would like to have that emotional change without feeling that they need to forgive, because I do believe that there are unforgivable offenses.

How do you facilitate forgiveness?

Where the science of forgiveness has been the most explored is actually in couples therapy. There is this clinician, Everett Worthington, who has a therapy called REACH with steps that people can follow. But it’s very different to forgive your spouse with whom you have had a relationship as opposed to, say, the commander of the battalion of the paramilitary group that killed your brother. You didn’t have a relationship with that person, and you don’t want to have a relationship later.

Contrary to popular belief, forgiveness is not exactly a situation that occurs in a relationship of two, a perpetrator and a victim. Two of my co-investigators in Colombia, Pablo Abitbol and Santiago Amaya, they think that forgiveness in postconflict Colombia actually is a three-party relationship that involves the community as well. It’s part of an ongoing investigation to try to fully understand: Does forgiveness have different concepts depending on the relationship? Is it the same phenomena? That’s what we’re trying to figure out.