maciek905/Getty ImagesI've been around technology for long enough that very little excites me, and even less surprises me. But shortly after Open AI's ChatGPT was released, I asked it to write a WordPress plugin for my wife's e-commerce site. When it did, and the plugin worked, I was indeed surprised.That was the beginning of my deep exploration into chatbots and AI-assisted programming. Since then, I've subjected 14 large machine models (LLMs) to four real-world tests. Also:The five biggest mistakes people make when prompting an AIUnfortunately, not all chatbots can code alike. It's been almost two years since that first test, and even now, five of the 14 LLMs I tested can't create working plugins.In this article, I'll show you how each LLM performed against my tests. There are two chatbots I recommend you use, but they cost $20/month. The free versions of the same chatbots do well enough that you could probably get by without paying. But the rest, whether free or paid, are not so great. I won't risk my programming projects with them or recommend that you do until their performance improves. I've written a lot about using AIs to help with programming. Unless it's a small, simple project, like my wife's plugin, AIs can't write entire apps or programs. But they excel at writing a few lines and are not bad at fixing code.Also: I tested DeepSeek's R1 and V3 coding skills - and we're not all doomed (yet)Rather than repeat everything I've written, go ahead and read this article:How to use ChatGPT to write code: What it can and can't do for you.If you want to understand my coding tests, why I've chosen them, and why they're relevant to this review of the 14 LLMs, read this article:How I test an AI chatbot's coding ability - and you can too. Let's start with a comparative look at how the chatbots performed: David Gewirtz/ZDNETNext, let's look at each chatbot individually. I'll discuss 13 chatbots, even though the above chart shows 14 LLMs. The results for GPT-4 and GPT-4o are both included in ChatGPT Plus. Ready? Let's go. David Gewirtz/ZDNET Pros Passed all testsSolid coding resultsMac app Cons HallucinationsNo Windows app yetSometimes uncooperative Price: $20/moLLM: GPT-4o, GPT-4, GPT-3.5Desktop browser interface: YesDedicated Mac app: YesDedicated Windows app: NoMulti-factor authentication: YesTests passed: 4 of 4ChatGPT Pluswith GPT-4 and GPT-4o passed all my tests. One of my favorite features is the availability of a dedicated app. When I test web programming, I have my browser set on one thing, my IDE open, and the ChatGPT Mac app running on a separate screen.Also:I put GPT-4o through my coding tests and it aced them - except for one weird resultIn addition,Logitech's Prompt Builder, which pops up using a mouse button, can be set up to use the upgraded GPT-4o and connect to your OpenAI account, making it a simple thumb-tap to run a prompt, which is very convenient.The only thing I didn't like was that one of my GPT-4o tests resulted in a dual-choice answer, and one of those answers was wrong. I'd rather it just gave me the correct answer. Even so, a quick test confirmed which answer would work. But that issue was a bit annoying. I didn't have that issue in GPT-4, so for now, that's the LLM setting I use with ChatGPT when coding. View now at OpenAI David Gewirtz/ZDNET Pros Multiple LLMsSearch criteria displayedGood sourcing Cons Email-only loginNo desktop app Price: $20/moLLM: GPT-4o, Claude 3.5 Sonnet, Sonar Large, Claude 3 Opus, Llama 3.1 405BDesktop browser interface: YesDedicated Mac app: NoDedicated Windows app: NoMulti-factor authentication: NoTests passed: 4 of 4I seriously considered listingPerplexity Proas the best overall AI chatbot for coding, but one failing kept it out of the top slot: how you log in. Perplexity doesn't use username/password or passkey, and doesn't have multi-factor authentication. All the tool does is email you a login pin. The AI also doesn't have a separate desktop app, as ChatGPT does for Macs.What sets Perplexity apart from other tools is that it can run multiple LLMs. While you can't set an LLM for a given session, you can easily go into the settings and choose the active model.Also:Can Perplexity Pro help you code? It aced my programming tests - thanks to GPT-4For programming, you'll probably want to stick to GPT-4o, because that aced all our tests. But it might be interesting to cross-check code across the different LLMs. For example, if you have GPT-4o write some regular expression code, you might consider switching to a different LLM to see what that LLM thinks of the generated code.As we'll see below, most LLMs are unreliable, so don't take the results as gospel. However, you can use the results to give you more things to check your original code. It's sort of like an AI-driven code review.Just don't forget to switch back to GPT-4o. View now at Perplexity.AI David Gewirtz/ZDNET Pros Different LLM than ChatGPTGood descriptionsFree access Cons Only available in browser modeFree access likely only temporary Price: Free (for now)LLM: Grok-1Desktop browser interface: YesDedicated Mac app: NoDedicated Windows app: NoMulti-factor authentication: YesTests passed: 3 of 4I have to say, Grok surprised me. I guess I didn't have high hopes for an LLM that appeared tacked onto the Social Network Formerly Known as Twitter. But then again, X is now owned by Elon Musk and two of Musk's companies, Tesla and SpaceX, have towering AI capabilities.It's not clear how much of the Tesla and SpaceX AI DNA went into Grok, but we can fairly assume that there will likely be more work. As it is now, Grok is the only LLM not based on OpenAI LLMs that made it into the recommended list.Also: X's Grok did surprisingly well in my AI coding testsGrok did make one mistake, but it was a relatively minor one that could be easily remedied by a slightly more comprehensive prompt. Yes, it failed the test. But by passing the others, and by even doing an almost perfect good job on the one it passed, it earned itself a spot as a contender.Stay tuned. This is one to watch. View now at X David Gewirtz/ZDNET Pros FreePassed most tests Cons Prompt throttlingCould cut you off in the middle of whatever you're working on Price: FreeLLM: GPT-4o, GPT-3.5Desktop browser interface: YesDedicated Mac app: YesDedicated Windows app: NoMulti-factor authentication: YesTests passed: 3 of 4 in GPT-3.5 modeChatGPTis available to anyone for free. While both the Plus and free versions support GPT-4o, which passed all my programming tests, there are limitations when using the free app.OpenAI treats free ChatGPT users as if they're in the cheap seats. If traffic is high or the servers are busy, the free ChatGPT will only make GPT-3.5 available to free users. The tool will only allow you a certain number of queries before it downgrades or shuts you off.Also:How to use ChatGPT: What you need to know nowI've had several occasions when the free version of ChatGPT effectively told me I'd asked too many questions.ChatGPT is a great tool, as long as you don't mind getting shut down sometimes. Even GPT-3.5 did better on the tests than all the other chatbots, and the test it failed was for a fairly obscure programming tool produced by a lone programmer in Australia.So, if budget is important to you and you can wait when cut off, go for ChatGPT free. View now at OpenAI David Gewirtz/ZDNET Pros FreePassed most testsRange of research tools Cons Limited to GPT-3.5Throttles prompt results Price: FreeLLM: GPT-3.5Desktop browser interface: YesDedicated Mac app: NoDedicated Windows app: NoMulti-factor authentication: NoTests passed: 3 of 4I'm threading a pretty fine needle here, but because Perplexity AI's free version is based on GPT-3.5, the test results were measurably better than the other AI chatbots.Also: 5 reasons why I prefer Perplexity over every other AI chatbotFrom a programming perspective, that's pretty much the whole story. But from a research and organization perspective, my ZDNET colleague Steven Vaughan-Nichols prefers Perplexity over the other AIs.He likes how Perplexity provides more complete sources for research questions, cites its sources, organizes the replies, and offers questions for further searches.So if you're programming, but also doing other research, consider the free version of Perplexity. View now at Perplexity.AI David Gewirtz/ZDNET Pros FreeOpen SourceEfficient resource utilization Cons Weak general knowledgeSmall ecosystemLimited integrations Price: Free for chatbot, fees for APILLM: DeepSeek MoEDesktop browser interface: YesDedicated Mac app: NoDedicated Windows app: NoMulti-factor authentication: NoTests passed: 3 of 4While DeepSeek R1 is the new reasoning hotness from China that has all the pundits punditing, the real power right now (at least according to our tests) is DeepSeek V3. This chatbot passed almost all of our coding tests, doing as well as the (now mostly discontinued) ChatGPT 3.5.Also: I tested DeepSeek's R1 and V3 coding skills - and we're not all doomed (yet)Where DeekSeek V3 fell down was in its knowledge of somewhat more obscure programming environments. Still, it beat out Google's Gemini, Microsoft's Copilot, and Meta's Meta AI, which is quite the accomplishment all on its own. We'll be keeping a close watch on each DeepSeek model, so stay tuned. View now at Deepseek Chatbots to avoid for programming helpI tested 14 LLMs, and seven passed most of my tests. The other chatbots, including a few pitched as great for programming, each only passed one of my tests -- and Microsoft's Copilot didn't pass any. I'm mentioning them here because people will ask, and I did test them thoroughly. Some bots do just fine for other work, so I'll point you to their general reviews if you're just curious about how they function. DeepSeek R1 David Gewirtz/ZDNETUnlike DeepSeek V3, the advanced reasoning version DeepSeek R1 did not showcase its reasoning capabilities when it came to our programming tests. It was odd that the new failure area was one that's not all that hard, even for a basic AI -- the regular expression code for our string function test.Also: I tested DeepSeek's R1 and V3 coding skills - and we're not all doomed (yet)But that's why we are running these real-world tests. It's never clear where an AI will hallucinate or just plain fail, and before you go believing all the hype about DeepSeek R1 taking the crown away from ChatGPT, run some programming tests. So far, while I'm impressed with the much reduced resource utilization and the open source nature of the product, its coding quality output is inconsistent.GitHub Copilot David Gewirtz/ZDNETGitHub's Copilot integrates quite seamlessly with VS Code. It makes asking for coding help very quick and productive, especially when working in context. That's why it's so disappointing that the code it writes can often be so very wrong.Also: I put GitHub Copilot's AI to the test - and it just might be terrible at writing codeI can't, in good conscience, recommend you use the GitHub Copilot extensions for VS Code. I'm concerned that the temptation will be too great to just insert blocks of code without sufficient testing -- and that GitHub Copilot's produced code is just not ready for production use. Try again next year. Meta AI David Gewirtz/ZDNETMeta AI is Facebook's general-purpose AI. As you can see above, it failed three of our four tests.Also: 15 ways AI saved me time at work in 2024 - and how I plan to use it in 2025The AI did generate a nice user interface but with zero functionality. And it did find my annoying bug, which is a fairly serious challenge. Given the specific knowledge required to find the bug, I was surprised it choked on a simple regular expression challenge. But it did. Meta Code Llama David Gewirtz/ZDNETMeta Code Llama is Facebook's AI designed specifically for coding help. It's something you can download and install on your server. I tested it running on a Hugging Face AI instance.Also:Can Meta AI code? I tested it against Llama, Gemini, and ChatGPT - it wasn't even closeWeirdly, even though both Meta AI and Meta Code Llama choked on three of four of my tests, they choked on different problems. AIs can't be counted on to give the same answer twice, but this result was a surprise. We'll see if that changes over time. Claude 3.5 Sonnet David Gewirtz/ZDNETAnthropic claims the 3.5 Sonnet version of its Claude AI chatbot is ideal for programming. After failing all but one test, I'm not so sure.If you're not using it for programming, Claude may be a better choice than the free version of ChatGPT.My ZDNET colleague Maria Diaz reports that Claude can handle uploaded files, process more words than the free version of ChatGPT, provide information roughly a year more current than GPT-3.5, and access websites. Gemini Advanced David Gewirtz/ZDNETGemini Advanced is Google's $20 pro version of its Gemini (formerly Bard) chatbot. I expected the tool to do better than one out of four. Interestingly, it passed the one test that every AI other than GPT-4/4o failed -- knowledge of that fairly obscure programming language produced by one programmer in Australia.So, if it knew that language, why couldn't it handle basic regular expressions or other first-year programming student problems? Microsoft Copilot David Gewirtz/ZDNETYou'd think the company with the "Developers! Developers! Developers!" mantra in its DNA would have an AI that does better on the programming tests. Microsoft produces some of the best coding tools on the planet. And yet, Copilot did badly.Also:What are Microsoft's different Copilots? Here are the differences and how you can use themThe one positive thing is that Microsoft always learns from its mistakes. So, I'll check back later and see if this result improves. But I like [insert name here]. Does this mean I have to use a different chatbot?Probably not. I've limited my tests to day-to-day programming tasks. None of the bots has been asked to talk like a pirate, write prose, or draw a picture. In the same way we use different productivity tools to accomplish specific tasks, feel free to choose the AI that helps you complete the task at hand.The only issue is if you're on a budget and are paying for a pro version. Then, find the AI that does most of what you want, so you don't have to pay for too many AI add-ons. It's only a matter of time The results of my tests were fairly surprising, especially given the big investments of Microsoft and Google. But this area of innovation is improving at warp speed, so we'll be back with updated tests and results over time. Stay tuned.Have you used any of these AI chatbots for programming? What has your experience been? Let us know in the comments below. You can follow my day-to-day project updates on social media. Be sure to subscribe to my weekly update newsletter, and follow me on Twitter/X at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.Featured