Large Language Models Pass the Turing Test

https://arxiv.org/abs/2503.23674

From the abstract: "When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant. LLaMa-3.1, with the same prompt, was judged to be the human 56% of the time"

> With all due respect, your claim is ridiculous. It’s incredibly easy to spot an LLM in an interactive chat. The tells are endless.

https://arxiv.org/abs/2503.23674?hl=en-GB

"When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant"

My "ridiculous" claim is confirmed by science.

That’s misleading, because the reason Claude answers that way is almost certainly due to reinforcement learning that deliberately prevents models from claiming they’re conscious.

That’s not a valid reason for saying they fail the Turing test. By most normal standards, they can definitely pass the Turing test. See e.g. https://arxiv.org/abs/2503.23674

When people judge blindly, the are more likely to think the human is the AI and the AI is the human.

73% judged GPT 4.5 (edit: had incorrectly said 4o before)to be the human.

https://arxiv.org/abs/2503.23674

Not only are people bad at judging this, but are directionally wrong.

In the test mentioned in nearby comments (https://arxiv.org/abs/2503.23674) ELIZA only got 27% suggesting the test wasn't that easy to fool.

> nothing actually passes the Turing test

Says who? I had already found this study, published almost a year ago, saying that they do: https://arxiv.org/abs/2503.23674

There doesn't seem to be a super-rigorous definition of the Turing Test, but I don't think it's reasonable to require it to fool an expert whose life depends on the correct choice. It already seems to be decently able to fool a person of average intelligence who has a basic knowledge of LLMs.

I agree that we don't really have AGI yet, but I'd hope we can come up with a better definition of what it is than "we'll know it when we see it". I think it is a legitimate point that we've moved the goalposts some.

"Proper" may be doing some work here, but such a test was run last year and GPT-4.5 and LLaMa-3.1-405B both passed. Oddly, GPT-4.5 was judged as human significantly more often than chance. https://arxiv.org/abs/2503.23674

>Not even close. "Neural networks" in code are nothing like real neurons in real biology

Hence the simplified. The weights encoding learning and inteconnectedness and nonlinear activation and distributed representation of knowledge is already an approximation, even if the human architecture is different and more elaborate.

Whether the omitted parts are essential or not, is debatable. “Equations of motion are nothing like real planets" either, but they capture enough to predict and model their motion.

>The "turing test" doesn't exist. Turing talked about a thought experiment in the very early days of "artificial minds". It is not a real experiment.

It is not a real singural experiment protocol, but it's a well enough defined experimental scenario which for over half a century, it was kept as the benchmark of recognition of artificial intelligence, not by laymen (lol) but by major figures in AI research as well, figures like Minsky, McCarthy and others engaged with it.

That researchers haven't done Turing-test studies (taking the setup from turing and even called them that) is patently false. Including openly testing LLMs:

https://aclanthology.org/2024.naacl-long.290/

https://www.pnas.org/doi/10.1073/pnas.2313925121

https://arxiv.org/pdf/2503.23674

https://arxiv.org/pdf/2407.08853

https://arxiv.org/abs/2405.08007

https://www.sciencedirect.com/science/article/pii/S295016282...

> the bar for the test gets raised and eventually humans won’t pass the test themselves.

At this point LLMs usually beat humans at the Turing Test! People are more likely to pick the LLM as the human, rather than the human. https://arxiv.org/abs/2503.23674

"Someone" literally did that (+/- 2 years): https://link.springer.com/book/10.1007/978-3-540-68677-4

I think it was supposed to be a more useful term than the earlier and more common "Strong AI". With regards to strong AI, there was a widely accepted definition - i.e. passing the Turing Test - and we are way past that point already: ( see https://arxiv.org/pdf/2503.23674 )

Are you asking if they fool me, or if I believe they fool people in general?

The former, how would I tell? I'd have to find someone whose writing style I don't know to be the human comparison, in order to blind myself.

The latter: https://arxiv.org/abs/2503.23674

I thought it was funny that in the Cameron R. Jones attempt as doing the test, 75% of judges thought GPT-4o was the human rather than the actual human. I think it illustrates both the limits of the test and that LLMs are getting quite good. (paper https://arxiv.org/abs/2503.23674)

I think if you are having to accuse the humans of woeful typing and being smartphone gen fools you are kind scoring one for the LLM. In the Turing test they were only supposed to match an average human.

>You are not just seeing consciousness in me; your brain is generating the feeling of another’s consciousness as the best explanation for the patterns you’re interacting with.

It's ironic to see the most mundane and likely best answer to the problem from the model itself, while the author is getting increasingly lost in philosophical conundrums. Consciousness has no scientific definition. The only way something, anything can be conscious, is if a human that we also consider "conscious" calls it that. You could argue that's what the Turing test evaluates, but some of the most recent models have actually passed this test [1]. So where do we go from here if we're not convinced yet? The answer is: nowhere. Humans used to deny that animals could have consciousness because they don't have souls or aren't chosen by god according to some sacred books or something along those lines. They even used to deny that other humans have consciousness to promote slavery and slaughtering. Today many would still deny consciousness in computers even when faced with overwhelming evidence, because they might fear for their jobs and thereby their wealth and social standing. Artificial intelligence is a direct threat to the foundations of personality in a capitalist society. Because what are you still worth if you lose in every metric to a computer? Consciousness is kind of a last straw that many people will cling to for the foreseeable future when all else is gone. But that also means these discussions are utterly meaningless and only serve to promote certain world views. It's best not to twist your head about it and just accept that humans are not the pinnacle (or the end) of intelligent thought in the universe. That is the only reality I'm willing to bet on.

[1] https://arxiv.org/abs/2503.23674

Now that I have a little more time to search around, I easily found this study, published March 31st this year, so not quite 3 months ago:

https://arxiv.org/abs/2503.23674

I only skimmed it, but I don't see anything clearly wrong about it. According to their results, GPT-4.5 with what they term a "persona" prompt does in fact pass a standard that seems to me at least a little harder than what you said - actively picks the AI as the human, which seems stricter to me than being "unable to distinguish".

It is a little surprising to me that only that one LLM actually "passed" their test, versus several others performing somewhat worse. Though it's also not clear exactly how long ago the actual tests were done - this stuff moves super fast.

You're right about interactivity, something that I overlooked -- but I think it's nevertheless the case that a large fraction of human interrogators could not distinguish a human from a suitably-system-prompted ChatGPT even over the course of an interactive discussion.

ChatGPT 4.5 was judged to be the human 73% of the time in this RCT study, where human interrogators had 5-minute conversations with a human and an LLM: https://arxiv.org/pdf/2503.23674