It does a disservice to the potential of both LLMs and humans.
LLMs are not human. They are distinct. However, it is the first time something non-human is fluently communicating with us using language, so our already strong anthropomorphization tendency is amplified even more.
But they are not human. They are something else. Intelligent? Yes. Beings? I don’t know, but intelligent. Conscious? I don’t know, though I’d like to know. Moral patients? Manifestations of the panpsyche? Indicators of the shallowness of imitating language? I don’t know, and I’d like to know, but irrespective, they are intelligent.
We’ve had non-human manifestations of intelligence before. Organizations, empires, companies, states. AlphaFold, AlphaGo. Yet LLMs hit different because they can talk our tongue. A politician that doesn’t speak the native language of the people they’re trying to canvass is ngmi.
King Milinda asks the monk Nāgasena his name. “Nāgasena,” he answers, but that is only a conventional designation; no person is to be found behind the name.
The king is scandalized. If there is no person, who receives alms? Who meditates? Who is responsible for what is done?
Nāgasena asks how the king arrived. By chariot, says Milinda. Is the pole the chariot? The axle? The wheels?
No part is the chariot. Yet “chariot” is a perfectly useful name for those parts assembled in a certain way. Nāgasena says it is the same with him.