www.technologyreview.com
When the Chinese firm DeepSeek dropped a large language model called R1 last week, it sent shock waves through the US tech industry. Not only did R1 match the best of the homegrown competition, it was built for a fraction of the costand given away for free.The US stock market lost $1 trillion, President Trump called it a wake-up call, and the hype was dialed up yet again. DeepSeek R1 is one of the most amazing and impressive breakthroughs Ive ever seenand as open source, a profound gift to the world, Silicon Valleys kingpin investor Marc Andreessen posted on X.But DeepSeeks innovations are not the only takeaway here. By publishing details about how R1 and a previous model called V3 were built and releasing the models for free, DeepSeek has pulled back the curtain to reveal that reasoning models are a lot easier to build than people thought. The company has closed the lead on the worlds very top labs.The news kicked competitors everywhere into gear. This week, the Chinese tech giant Alibaba announced a new version of its large language model Qwen and the Allen Institute for AI (AI2), a top US nonprofit lab, announced an update to its large language model Tulu. Both claim that their latest models beat DeepSeeks equivalent.Sam Altman, cofounder and CEO of OpenAI, called R1 impressivefor the pricebut hit back with a bullish promise: We will obviously deliver much better models. OpenAI then pushed out ChatGPT Gov, a version of its chatbot tailored to the security needs of US government agencies, in an apparent nod to concerns that DeepSeeks app was sending data to China. Theres more to come.DeepSeek has suddenly become the company to beat. What exactly did it do to rattle the tech world so fully? Is the hype justified? And what can we learn from the buzz about whats coming next? Heres what you need to know.Training stepsLets start by unpacking how large language models are trained. There are two main stages, known as pretraining and post-training. Pretraining is the stage most people talk about. In this process, billions of documentshuge numbers of websites, books, code repositories, and moreare fed into a neural network over and over again until it learns to generate text that looks like its source material, one word at a time. What you end up with is known as a base model.Pretraining is where most of the work happens, and it can cost huge amounts of money. But as Andrej Karpathy, a cofounder of OpenAI and former head of AI at Tesla, noted in a talk at Microsoft Build last year: Base models are not assistants. They just want to complete internet documents.Turning a large language model into a useful tool takes a number of extra steps. This is the post-training stage, where the model learns to do specific tasks like answer questions (or answer questions step by step, as with OpenAIs o3 and DeepSeeks R1). The way this has been done for the last few years is to take a base model and train it to mimic examples of question-answer pairs provided by armies of human testers. This step is known as supervised fine-tuning.OpenAI then pioneered yet another step, in which sample answers from the model are scoredagain by human testersand those scores used to train the model to produce future answers more like those that score well and less like those that dont. This technique, known as reinforcement learning with human feedback (RLHF), is what makes chatbots like ChatGPT so slick. RLHF is now used across the industry.But those post-training steps take time. What DeepSeek has shown is that you can get the same results without using people at allat least most of the time. DeepSeek replaces supervised fine-tuning and RLHF with a reinforcement-learning step that is fully automated. Instead of using human feedback to steer its models, the firm uses feedback scores produced by a computer.Skipping or cutting down on human feedbackthats a big thing, says Itamar Friedman, a former research director at Alibaba and now cofounder and CEO of Qodo, an AI coding startup based in Israel. Youre almost completely training models without humans needing to do the labor.Cheap laborThe downside of this approach is that computers are good at scoring answers to questions about math and code but not very good at scoring answers to open-ended or more subjective questions. Thats why R1 performs especially well on math and code tests. To train its models to answer a wider range of non-math questions or perform creative tasks, DeepSeek still has to ask people to provide the feedback.But even that is cheaper in China. Relative to Western markets, the cost to create high-quality data is lower in China and there is a larger talent pool with university qualifications in math, programming, or engineering fields, says Si Chen, a vice president at the Australian AI firm Appen and a former head of strategy at both Amazon Web Services China and the Chinese tech giant Tencent.DeepSeek used this approach to build a base model, called V3, that rivals OpenAIs flagship model GPT-4o. The firm released V3 a month ago. Last weeks R1, the new model that matches OpenAIs o1, was built on top of V3.To build R1, DeepSeek took V3 and ran its reinforcement-learning loop over and over again. In 2016 Google DeepMind showed that this kind of automated trial-and-error approach, with no human input, could take a board-game-playing model that made random moves and train it to beat grand masters. DeepSeek does something similar with large language models: Potential answers are treated as possible moves in a game.To start with, the model did not produce answers that worked through a question step by step, as DeepSeek wanted. But by scoring the models sample answers automatically, the training process nudged it bit by bit toward the desired behavior.Eventually, DeepSeek produced a model that performed well on a number of benchmarks. But this model, called R1-Zero, gave answers that were hard to read and were written in a mix of multiple languages. To give it one last tweak, DeepSeek seeded the reinforcement-learning process with a small data set of example responses provided by people. Training R1-Zero on those produced the model that DeepSeek named R1.Theres more. To make its use of reinforcement learning as efficient as possible, DeepSeek has also developed a new algorithm called Group Relative Policy Optimization (GRPO). It first used GRPO a year ago, to build a model called DeepSeekMath.Well skip the detailsyou just need to know that reinforcement learning involves calculating a score to determine whether a potential move is good or bad. Many existing reinforcement-learning techniques require a whole separate model to make this calculation. In the case of large language models, that means a second model that could be as expensive to build and run as the first. Instead of using a second model to predict a score, GRPO just makes an educated guess. Its cheap, but still accurate enough to work.A common approachDeepSeeks use of reinforcement learning is the main innovation that the company describes in its R1 paper. But DeepSeek is not the only firm experimenting with this technique. Two weeks before R1 dropped, a team at Microsoft Asia announced a model called rStar-Math, which was trained in a similar way. It has similarly huge leaps in performance, says Matt Zeiler, founder and CEO of the AI firm Clarifai.AI2s Tulu was also built using efficient reinforcement-learning techniques (but on top of, not instead of, human-led steps like supervised fine-tuning and RLHF). And the US firm Hugging Face is racing to replicate R1 with OpenR1, a clone of DeepSeeks model that Hugging Face hopes will expose even more of the ingredients in R1s special sauce.Whats more, its an open secret that top firms like OpenAI, Google DeepMind, and Anthropic may already be using their own versions of DeepSeeks approach to train their new generation of models. Im sure theyre doing almost the exact same thing, but theyll have their own flavor of it, saysZeiler.But DeepSeek has more than one trick up its sleeve. It trained its base model V3 to do something called multi-token prediction, where the model learns to predict a string of words at once instead of one at a time. This training is cheaper and turns out to boost accuracy as well. If you think about how you speak, when youre halfway through a sentence, you know what the rest of the sentence is going to be, says Zeiler. These models should be capable of that too.It has also found cheaper ways to create large data sets. To train last years model, DeepSeekMath, it took a free data set called Common Crawla huge number of documents scraped from the internetand used an automated process to extract just the documents that included math problems. This was far cheaper than building a new data set of math problems by hand. It was also more effective: Common Crawl includes a lot more math than any other specialist math data set thats available.And on the hardware side, DeepSeek has found new ways to juice old chips, allowing it to train top-tier models without coughing up for the latest hardware on the market. Half their innovation comes from straight engineering, says Zeiler: They definitely have some really, really good GPU engineers on that team.Nvidia provides software called CUDA that engineers use to tweak the settings of their chips. But DeepSeek bypassed this code using assembler, a programming language that talks to the hardware itself, to go far beyond what Nvidia offers out of the box. Thats as hardcore as it gets in optimizing these things, says Zeiler. You can do it, but basically its so difficult that nobody does.DeepSeeks string of innovations on multiple models is impressive. But it also shows that the firms claim to have spent less than $6 million to train V3 is not the whole story. R1 and V3 were built on a stack of existing tech. Maybe the very last stepthe last click of the buttoncost them $6 million, but the research that led up to that probably cost 10 times as much, if not more, says Friedman. And in a blog post that cut through a lot of the hype, Anthropic cofounder and CEO Dario Amodei pointed out that DeepSeek probably has around $1 billion worth of chips, an estimate based on reports that the firm in fact used 50,000 Nvidia H100 GPUs.A new paradigmBut why now? There are hundreds of startups around the world trying to build the next big thing. Why have we seen a string of reasoning models like OpenAIs o1 and o3, Google DeepMinds Gemini 2.0 Flash Thinking, and now R1 appear within weeks of each other?The answer is that the base modelsGPT-4o, Gemini 2.0, V3are all now good enough to have reasoning-like behavior coaxed out of them. What R1 shows is that with a strong enough base model, reinforcement learning is sufficient to elicit reasoning from a language model without any human supervision, says Lewis Tunstall, a scientist at Hugging Face.In other words, top US firms may have figured out how to do it but were keeping quiet. It seems that theres a clever way of taking your base model, your pretrained model, and turning it into a much more capable reasoning model, says Zeiler. And up to this point, the procedure that was required for converting a pretrained model into a reasoning model wasnt well known. It wasnt public.Whats different about R1 is that DeepSeek published how they did it. And it turns out that its not that expensive a process, says Zeiler. The hard part is getting that pretrained model in the first place. As Karpathy revealed at Microsoft Build last year, pretraining a model represents 99% of the work and most of the cost.If building reasoning models is not as hard as people thought, we can expect a proliferation of free models that are far more capable than weve yet seen. With the know-how out in the open, Friedman thinks, there will be more collaboration between small companies, blunting the edge that the biggest companies have enjoyed. I think this could be a monumental moment, he says.
