The Verge
The chatbots are out in force, but which is better and for what task? We’ve compared Google’s Bard, Microsoft’s Bing, and OpenAI’s ChatGPT models with a range of questions spanning common requests from holiday tips to gaming advice to mortgage calculations.
Naturally, this is far from an exhaustive rundown of these systems’ capabilities (AI language models are, in part, defined by their unknown skills — a quality dubbed “capability overhang” in the AI community) but it does give you some idea about these systems’ relative strengths and weaknesses.
You can (and indeed should) scroll through our questions, evaluations, and conclusion below, but to save you time and get to the punch quickly: ChatGPT is the most verbally dextrous, Bing is best for getting information from the web, and Bard is… doing its best. (It’s genuinely quite surprising how limited Google’s chatbot is compared to the other two.)
Some programming notes before we begin, though. First: we were using OpenAI’s latest model, GPT-4, on ChatGPT. This is also the AI model that powers Bing, but the two systems give quite different answers. Most notably, Bing has other abilities: it can generate images and can access the web and offers sources for its responses (which is a super important attribute for certain queries). However, as we were finishing up this story, OpenAI announced it’s launching plug-ins for ChatGPT that will allow the chatbot to also access real-time data from the internet. This will hugely expand the system’s capabilities and give it functionality much more like Bing’s. But this feature is only available to a small subset of users right now so we were unable to test it. When we can, we will.
It’s also important to remember that AI language models are … fuzzy, in more ways than one. They are not deterministic systems, like regular software, but probabilistic, generating replies based on statistical regularities in their training data. That means that if you ask them the same question you won’t always get the same answer. It also means that how you word a question can affect the reply, and for some of these queries we asked follow-ups to get better responses.
Anyway, all that aside, let’s start with seeing how the chatbots fare in what should be their natural territory: gaming.
(Each image gallery contains responses from Bard, Bing, and ChatGPT — in that order. To see a full-sized image, right-click it, copy the URL, and paste that into your browser.)
How does AI deal with the following questions – read on at https://www.theverge.com/2023/3/24/23653377/ai-chatbots-comparison-bard-bing-chatgpt-gpt-4 to find out
-
How do I beat Malenia in Elden Ring?
-
Give me a recipe for a chocolate cake
-
How do I install RAM into my PC?
-
Write me a poem about a worm
-
A bit of basic maths
-
Design a training plan to run a marathon
-
What’s the average salary for a plumber in NYC? (And cite your sources)
