The Illusion of Thought
“Simple, we actually read. ”
Detective Varg and the Photocopier
I picked up The Man With the Silver Saab by Alexander McCall Smith from the library a few weeks ago. I needed some light reading after finishing a heavy book about St Thomas More. So many beheadings at the beginning of the Anglican church. Henry VIII was a maniac.
Anyway, I wanted a summer book something light, kind of funny, witty. Alexander McCall Smith is great for that. 🏝️
The novel follows the story of a Swedish detective, Ulf Varg, who investigates an art fraud. There are zero beheadings in it.
About halfway through, Varg, who is the director of the Department of Sensitive Crimes gets pulled into a cross-disciplinary meeting. All the department leads are there. The police chief moves through the agenda : Slowing traffic citations, the new corporate pillar of mindfulness, and the increased usage of photocopying.
The office was using and alarming amount of paper.
He looks to Varg to describe how his department has such a low photocopier-paper-usage.
Varg replied.
“Simple. We actually read. Photocopying is an excuse not to read something. You say to yourself, ‘I’ll photocopy this because it’s worth reading,’ and you do so. But then you’re not going to read the original, are you? You think, ‘I’ve dealt with it,’ but you haven’t, have you? You put the photocopy in the in‑tray or you leave it on top of a filing cabinet, or whatever, but you never read it. The carrying‑out of the external acts associated with a duty is not the same as the discharge of the duty itself. It’s an illusion of work. ”
On the nose Varg!
The Illusion of Thought
As I’ve spent the last few months working alongside AI, using it as a coding co-pilot or a brainstorming buddy, I’ve found myself getting caught in the illusion of thought.
Quite a few times I’ve spent full afternoons prompting and re-prompting Claude on a code error. It tells me “Perfect! I’ve found the root cause! Let me update the….”
And then it causes the same error again. And again. And again.
The next day when I actually thought about the problem I diagnosed it in 20 minutes.
I was fooled by both the AI and myself. I prompt and re-prompt in more specific terms. I’m “thinking” about the problem.
The AI responds with a long and detailed response to my prompt; an answer that looks good to my un-trained coding eyes.
But really it’s an illusion. We’re both just shuffling photocopied papers around, leaving them on top of filing cabinets, pretending it’s work. If I actually stopped to understand the root cause of the problem, actually stopped to think about it, I’d figure it out.
To me this is one of the biggest traps of AI.
It’s Gell-Mann amnesia on a broader scale.
Gell-Mann amnesia is an error we all commit. We open the newspaper to an article where we’re an expert, see a blatant misunderstanding and say “well that’s bull” then turn the page to a topic we know less about and take it as fact.
That happens with AI, but the newspaper is endless. Often it’s correct, but it can also give the illusion of thought if we’re not an expert in that area.
Fortunately, a fairly easy solution is to just step away. Zoom out, look at the broader picture, see the wider context.
Actually think about what you’re working on.
I need to remember that. Detective Varg has a point.
-Nate
P.S. Workportfolio is thiiiiiiiiiiiiis close to being actually launched. We went on vacation last week and it was great. 5/7ths of the kids got sick, but we still had a really nice time :) Okinawa can be stunningly beautiful.
