I was a little sloppy in my formulation. A better way would be the ability to observe one's own thought processes and integrate them back into consciousness. Meta-cognition is a much better term, I think. It doesn't seem to me like cats for instance have this ability.
You should check out the YouTube of the cat that was trained to use buttons to speak. It sometimes says things that are surprising which it wasn't trained to say directly - like complaining that its owners music is bad. I realize that this isn't contradicting your thesis of language models not being conscious - but I just generally think people should check out things like that cat. So I'm not trying to disagree with you here. I just think the cat is cool.
I'm also a fan of ants. They have agriculture. They farm in their underground cities! They keep livestock too; insect livestock, but still! They give them medicines when they get sick, specially cultivated fungus for example. They are so cool.
I already have, and I'm considering trying it out with my own cat at some point. Cats are way more intelligent than people give them credit for, I agree. But I think you can get very far without the ability to turn the mind's eye inward, and I don't think it's controversial to state that cats don't have a great deal of self-awareness and metacognitive ability.
Neosocial insects like ants are amazing indeed. I often wonder what kind of "mind" an anthill has. Hurts just to think about it.
And, to quote Oliver Sachs: "I'm not sure about octopuses..."
A way to make it stop hurting to think about, replaced instead by transcendental beauty, is to abstract pheromones to outcome utility backward induction and the following of pheromones to a single worker being a monte carlo tree search rollout with the pheromones as a guide: they play a mixed strategy proportional to the pheromones. The fading of pheromones becomes something like a weighted moving average. Strong parallels with game theory and reinforcement learning algorithms start to emerge. This model is wrong, but the Earth is also not a perfect sphere.
I find it very fun to think about and have been called wierd for taking about ants before with childlike excitement.
The AntsCanada channel is fun for watching ecosystems interact with each other. So fascinating. :)
Oliver said this during a roundtable discussion on metaphysics with Daniel Dennet, Steve Gould, Rupert Sheldrake, Freeman Dyson, and Stephen Toulmin from 1993.
It's a fascinating discussion with a lot of my favourite mid-late 20th century thinkers. It's up on youtube, lookup any of the names involved with "roundtable discussion". It's over 3 hours long though.
Anyway, Oliver was talking about what kind of mind he imagined various animals having. And that was most of what he had to say on octopuses.
It's a bit of a sad watch too because all the participants except Dennet and Sheldrake are dead now.
And yet, most would agree that a cat can feel pain, but a machine can't. I think most people would object if the Google engineer said "I think this AI is every bit as sentient as a cat." They'd still say he's crazy.
I also wonder what would happen if that particular AI actually had logic to "observe its own thought processes and integrate them back into consciousness". Would we then just spin on the semantics, such as by arguing that this isn't really "observing" and it certainly isn't "consciousness"? (how can you define words like "consciousness" without using words or concepts like "sentience", bringing us back to square one?)
So I don't think meta-cognition is the answer either.
