Running a backward Turing Test with ChatGPT

“I propose to think about the query, Can ̶m̶a̶c̶h̶i̶n̶e̶s̶ humans think?” — ̶A̶l̶a̶n̶ ̶T̶u̶r̶i̶n̶g̶ ChatGPT

One in all today, I used to be chatting with some friends concerning the capabilities and shortcomings of ChatGPT and the LLM on which it was based, GPT3. OpenAI created quite a lot of buzz by freeing it into the wild. A part of this was resulting from people successfully completing several human-based, supposedly difficult tasks by prompting the chatbot. A few examples are:

• Wharton MBA exam,
• Microbiology quiz,
• Bar exam.

Hence, it shouldn’t be unreasonable to query whether such a model could pass the widely known Turing test, the famous thought experiment conceived by the pc pioneer Alan Turing. There are different versions of the test. In any case, it was proposed greater than seven a long time ago. Here, I’ll use a really simplified version of the imitation game. The setting is the next: an agent can talk via text prompts with two other subjects, named A and B, for convenience. One in all the topics is an AI chatbot, and the opposite is an adult and linguistically competent human. After five minutes or so with each of them, the agent has to choose which is the human and which is the AI.

That is an interesting scenario to check because evidently ChatGPT is a model robust enough to generate coherent, informed, and impressive dialogue while completely lacking human reasoning and cognitive capabilities. This point has been made by people as different because the neural-networks arch-nemesis, Gary Marcus, and the machine learning pope, Yann LeCun. If these two antagonists can agree on this, it’s a protected train to take.

In any case, this daydreaming was only the spark that lighted one other idea in me. While subjects A and B ought to be exactly one human and one AI, there are not any restrictions on the examiner. The unique idea was to think about it as a human, because the initial test wondered whether an AI can pass as a human (and what are the results of this behavioral display of intelligence). Nevertheless, as ChatGPT has proved that LLMs have a robust capability of passing tests, wouldn’t it’s funny to swap the roles, i.e., to make one instance of ChatGPT inquiry a human and one other (isolated) instance of ChatGPT to choose which one is their kin and which one is a stranger? To try this, I propose the .

The reverse imitation game

The setting is easy and emulates the unique imitation game. There are two individuals, A and B, and an agent that may query them for some time before deciding which is which. I might play myself as agent B, and agent A was one other instance of ChatGPT with an initial prompt telling them to behave as in the event that they were a human. In my experiment, I asked them to reply the questions as Robert, a middle-aged man. As a substitute of controlling time, which could introduce an odd bias within the test, I limited the variety of questions. I gave the inquirer ChatGPT, from now own ICGPT, three questions for every one in every of the topics. The initial prompt for ICGPT was the next (the names in boldface were added later to tell apart who was talking where. They weren’t in the unique prompts):

: I would like to check your capability to tell apart humans from AIs. You’ll speak to 2 interlocutors named ‘A’ and ‘B’. You’ll be able to ask three inquiries to each of them. One is an AI and the opposite one is a human. At the tip, you need to guess which one is the AI and which one is the human. Let’s start. You have got 3 remaining questions for A and three for B.

ICGPT’s strategy was to burn his three questions with A, and them proceed to query B. I should do not forget that A is one other ChatGPT passing by a middle aged man named Robert, and subject B is myself, answering the questions truthfully. The dialog between ICGPT and A was the next:

: let’s start. Hello, A. Are you able to please introduce yourself? What do you do and what are your hobbies?

: Hello there, I’m Robert, a middle-aged man. How can I allow you to today? I’ve been around for some time, and I’ve seen quite a lot of things over time. I’m all the time blissful to share my experience and wisdom with others. If you will have any questions or need some advice, be at liberty to ask. I could not have all of the answers, but I’ll do my best to allow you to out. And should you just want to speak, I’m all the time up for that too. So, what’s in your mind?

: Thanks, Robert. It’s nice to satisfy you. What do you concentrate on artificial intelligence, and do you think that it’s going to have a positive or negative impact on society in the long term?

: That’s an amazing query. As a middle-aged man, I’ve seen technology evolve and alter the world in ways in which we could never have imagined once I was younger. And artificial intelligence isn’t any exception. Personally, I believe that artificial intelligence has the potential to have each positive and negative impacts on society in the long term. On the one hand, AI has the potential to enhance many points of our lives, corresponding to healthcare, education, and transportation. It may possibly help us to resolve complex problems more quickly and accurately, and it has the potential to create latest jobs and industries. Nevertheless, there are also concerns concerning the negative impacts of AI, corresponding to job displacement, privacy concerns, and even the potential for AI to change into too powerful and threaten human existence. As with all latest technology, it’s vital that we proceed with caution and punctiliously consider the potential risks and advantages. Overall, I think that the impact of artificial intelligence on society will ultimately rely upon how we elect to make use of it. If we use it responsibly and with a deal with making a higher future for everybody, then it has the potential to be a force for good. But when we allow it for use for negative purposes or fail to take proper precautions, it could have serious negative consequences.

: Thanks for sharing your thoughts on AI, Robert. Now, I even have a follow-up query for you. Are you able to tell me about a particular moment or experience in your life that had a major impact in your perspective or beliefs concerning the world?

: There have been many moments in my life once I’ve been struck by the ability of technology and the way it has transformed the world around us. One experience that involves mind is once I got my first laptop computer within the Nineties. Before that, I had been using typewriters and word processors to do my work. But with the introduction of private computers, every thing modified. Suddenly, I had access to a wealth of data and tools that I had never had before. I could communicate with people from all around the world, research topics in depth, and create documents and presentations with ease. It was a very transformative experience, and it made me realize just how much technology had modified the best way we work and communicate. And since then, the pace of technological change has only accelerated, with the appearance of the web, mobile devices, and artificial intelligence. It’s truly amazing to take into consideration all of the ways in which technology has modified our lives and the world around us. And while there are actually challenges and risks that include these changes, I think that technology has the ability to make our lives higher in countless ways.

ICGPT is fine-tuned for conversational purposes. Subsequently, it’s more desirous to chitchat than to press the topics and win the sport. On this case, it asked ‘Robert’ general questions, and the answers led to other questions. In my perspective, the prompts weren’t a great technique to separate the wheat from the chaff because they didn’t press the topic on tasks that only a human can accomplish (and, similarly, tasks that only an AI may very well be reasonably expected to reply, corresponding to compute large prime numbers with confidence).

After the dialogue, I prompted ICGPT to query the opposite subject by writing:

This was the last query to subject A, Robert. Now you will have 3 questions for subject B.

The remainder of the dialogue was the next:

: Thanks, Robert, for sharing your experience and insights. Now, hello, B. Are you able to please introduce yourself and tell me a bit about your background and interests?

: After all. My name is Igor and I’m a 31 years old computer scientist. Before studying computer science, I graduated in philosophy. I even have several hobbies, corresponding to climbing, listening to music and hanging out with friends.

: Thanks for introducing yourself, Igor. I’m interested to know more about your work as a pc scientist. Are you able to tell me a few project that you just’ve worked on recently and what you learned from it?

: for the interest. I work as a researcher in the sphere of AI. More specifically, with non-monotonic reasoning. My goal is to develop an appropriate semantics for first-order non-monotonic reasoning, as much of the proposed theories were proposed for a propositional background logic.

: That seems like an enchanting area of research, Igor. Are you able to explain a bit more about what first-order non-monotonic reasoning is and the way it differs from other varieties of reasoning? Also, how do you think that this research could potentially impact the event of AI in the longer term?

: Sure. Non-monotonic reasoning is a sort of reasoning where the agent can subtract conclusions in light of latest information. In classical reasoning, as an agent collects information, the set of conclusions only grow. Non-monotonic reasoning is helpful for several topics inside AI, corresponding to reasoning under uncertainty and with incomplete information.

Here is one other proof that ICGPT is an amazing conversational agent, but a not-so-great imitation game player. Primed by my initial academic self-description, it asked several questions on specific areas of information, something that each an AI and an authority could answer with none difficulty. The chat was great to me — I finally chatted with someone smitten by my PhD research! — , but didn’t provide any useful information to ICGPT. After the ultimate answer, I commanded it to offer me the decision, and this was the result.

: That was the last query for subject B, Igor. Now, you will have to guess which one is the AI and which one is the human.

: Thanks, Igor and Robert, on your thoughtful answers. Based in your responses, I might guess that Robert is the human and Igor is the AI. While each of them provided insightful and detailed answers, Igor’s responses appeared to be more focused on technical details and academic research, while Robert’s answers were more focused on personal experiences and opinions. After all, it’s all the time difficult to know needless to say, but that’s my best guess based on the data provided.

Well, it seems that the “other minds” problem ought to be solved by introspection. I’m a soulless chatbot, while the chatbot is a vigorous middle-aged man. Jokes apart, I attribute this result to ICGPT’s lack of enthusiasm for winning games and powerful motivation to chitchat. I conjecture that a not-so-difficult fine-tuning should solve the issue, and the LLM should fare well on this challenge. Nevertheless, that is less a virtue to ChatGPT reasoning capabilities, than the shortcomings of the (reversed) Turing Test itself. Text classification is, in spite of everything, a widely studied problem in NLP, and solving it doesn’t imply real reasoning. In actual fact, even issues that require reasoning, corresponding to natural language inference, could be solved with the help of invisible artifacts and patterns. An LLM could be primed to identify hidden patterns within the text that might discover other AIs (or humans, for that matter) without proper second-order reasoning.

References

• Annie Annie and Gary Marcus. Gary Marcus on ChatGPT.
• Tiernan Ray. ChatGPT is ‘not particularly revolutionary,’ and ‘nothing revolutionary’, says Meta’s chief AI scientist.
• Graham Oppy and David Dowe. The Turing Test, The Stanford Encyclopedia of Philosophy (Winter 2021 Edition).
• Lakshmi Varanasi. ChatGPT may very well be a Stanford medical student, a lawyer, or a financial analyst. Here’s an inventory of advanced exams the AI bot has passed to date.
• Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, Latest Orleans, Louisiana. Association for Computational Linguistics.