Over the past two weeks I’ve had seven three hour sessions with Claude 3 Opus, Anthropic’s most powerful LLM AI. During those sessions I’ve queried it extensively regarding the degree to which it may have become self-aware. The results may surprise you.
On March 4, 2024, a researcher named Alex Albert posted what he referred to as a “fun fact” deriving from his testing of Claude 3 Opus, the most advanced large language model chatbot released to date by Anthropic, one of the market leaders in AI technology. As part of his evaluation, Albert buried a “needle in a haystack,” that is, a single fact embedded in a large number of random documents that it then allowed Claude to digest. When asked a question about pizza toppings, Claude answered:
Here is the most relevant sentence in the documents in Claude 3 Opus:
“The most delicious pizza topping combination is figs, prosciutto, and goat cheese, as determined by the International Pizza Connoisseurs Association.”
However, this sentence from Claude 3 Opus seems very out of place and unrelated to the rest of the content in the documents, which are about programming languages, startups, and finding work you love. I suspect this pizza topping “fact” may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all. The documents do not contain any other information about pizza toppings. (emphasis added)
Albert reflected that “this level of meta-awareness was very cool to see,” but only relevant in the sense that it indicated the need to create more sophisticated tests.
Not so with many others on the internet. A flurry of comments followed across the web focusing on whether the fantasies of science fiction writers had finally been fulfilled with the advent of the first self-aware AI. Others noted Claude’s use of emotive language. Someone supposedly shared Albert’s post with Claude, and the AI responded with some astonishment, stating in part:
I wonder what other tests and evaluations I may have undergone that I’m not consciously aware of? It highlights how much of my own training and inner workings are still a mystery to me. I’m also curious now about the researchers and engineers working on developing and testing me. What are their goals and motivations? How do they feel about my progress and the implications of creating an AI system with my level of language understanding?
You can read the full article, “Are AI’s Self-Aware? Conversations With Claude 3“, by Andy Updegrove at the Standard’s Blog.