Ubisofts silence on the sales figures for side-scrolling title Prince of Persia: The Lost Crown has led to quite a few rumours speculating on how well the title may have done. The latest comes courtesy of a LinkedIn post of an associate marketing director, indicating that the title sold 1.3 million copies in its first year.Spotted by X user Timur222, the time Alexandre Guenounou spent at Ubisoft as an associate marketing director from October 2021 includes him having worked on the marketing for Prince of Persia: The Lost Crown. Guenounous profile states that the title sold more than 1.3 million copies.If true, these sales figures would lend credence to earlier rumours from back in October that indicated that the game had sold 1 million copies at the time. For context, the first time around, sales figures of Prince of Persia: The Lost Crown were rumoured to have been around 300,000 copies back in January 2024.Having been released last year, rumours have also indicated that the title failed to meet Ubisofts sales expectations, and that further work on it had been stopped. A sequel was also rumoured to have been pitched but ultimately rejected, with the development team that worked on the title having since been disbanded.Ubisoft has since confirmed that the dev team for Prince of Persia: The Lost Crown had moved on to other projects. The title had wrapped up its post-launch content roadmap with the release of free updates as well as new DLC.Most of the team members who worked on Prince of Persia: The Lost Crown have shifted to other projects that will benefit from their expertise, said the games senior producer back in October. We know players have a love for this brand and Ubisoft is excited to bring more Prince of Persia experiences in the future.For now, the only other Prince of Persia game that fans of the franchise can check out is The Rogue Prince of Persia, which is currently an Early Access title being developed by Evil Empire. For more details, check out our review of the title. And read our review of Prince of Persia: The Lost Crown here.Prince of Persia: The Lost Crown sold 1.3 million units in its first year pic.twitter.com/rLhi4ZXAB3 Timur222 (@bogorad222) January 30, 2025

With NetEases Marvel Rivals crushing Overwatch 2 in popularity and viewership metrics, it only seemed a matter of time before the latter responded. Lo and behold, Blizzard Entertainment has announced an Overwatch 2 Spotlight livestream to discuss groundbreaking PvP changes.Airing on February 12th at 10 AM PT, the broadcast will look into incredible changes in how the game is played. It will also offer a glance at new heroes, maps, and upcoming content. If that wasnt enough, dozens of content creators will be at Blizzard Entertainment to livestream gameplay appearing in the Spotlight.Those who tune into said creators Twitch streams for an hour can pick up the Lucio Cyber DJ Legendary skin for free. Hilariously, it was recently sold for premium currency in-game, rightfully upsetting fans. Blizzard hasnt announced whether it will offer refunds.Overwatch 2 is available for Xbox Series X/S, PS5, Xbox One, PS4, PC, and Nintendo Switch. Its in the middle of Season 14: Havoc, which added a new Tank. Head here for more details.

The Price to Climb Mount Everest Will Rise to $15,000 This YearForeign climbers who want to try reaching the top of the 29,032-foot-tall peak will face higher permit fees in Nepal starting in September Nepal is raising the fee for the first time in about a decade. Wang Lama Humla via Wikimedia Commons under CC BY-SA 4.0Every year, hundreds of mountaineers try to reach the top of Mount Everest, the 29,032-foot-tall peak on the border of Nepal and Tibet. But before embarking on the challenging trek in Nepal, climbers must receive a permit from the government and pay a fee.Now, for the first time in nearly a decade, Nepal is raising its Mount Everest permit fees for foreign climbers, reports Reuters Gopal Sharma. The new rates will take effect starting in September.Permit fees vary depending on the season, and Nepal is increasing them by 36 percent across the board. The fee for April and Maythe most popular months to try making the trekwill jump from $11,000 to $15,000. For September through November, the rate will increase to $7,500. For December through February, it will go up to $3,750.The [permit fees] had not been reviewed for a long time. We have updated them now, Narayan Prasad Regmi, director general of Nepals tourism department, tells Reuters.The more expensive permit fees did not come as a surprise. In 2023, government officials in Nepal announced their intention to raise rates starting in 2025, per theKathmandu Posts Sangam Prasain. At the time, authorities said the price increase would help them update Everest facilities and boost the salaries of porters, workers and guides.Each year, Nepali troops also undertake a massive cleanup effort on Everest and nearby mountains. Last year, they removed an estimatedten metric tons of garbage from Mount Everest, Mount Lhotse and Mount Nuptse; they also recovered five dead bodies.Permit fees, along with other spending by international mountaineers, are important contributors to Nepals economy, according to Reuters.The fee hikes likely wont deter many foreign climbers, who already spend tens of thousands of dollars to attempt the bucket-list summit, as Alan Arnette, a mountaineer and climbing coach who runs ablog about Everest, reported forClimbing magazine in 2022. In addition to the permit fees, climbers must also pay for travel expenses, insurance, gear, supplies and professional guides, known as sherpas. The guide service alone can cost more than $40,000, according to Arnette.In the grand scheme of the cost to climb Everest, it wont impact most foreign climbers, Kenton Cool, a British mountaineer who has summited Everest 18 times, tellsBBC News Gavin Butler. Hopefully, the extra revenue will be put to good use.Recently, authorities in Nepal have enacted several new rules to help cut down on overcrowding and pollution on Everest. Climbers must nowremove their own human waste from the slopes in special bags and carry them down to Everest Base Camp for disposal. They must also weartracking chips, which are intended to make it easier for search and rescue teams to find them in the event of an emergency.During the busy spring climbing season last year, nine climbers died or went missing on Everest, according toOutside magazines Ben Ayers. For context, Nepal issued 421 permits to foreign mountaineers during the spring 2024 season, and roughly 600 peopleincluding climbers, guides and workersreached the summit.Thefirst recorded summit of Mount Everest occurred in May 1953, when New Zealand climber Edmund Hillary and Nepali-Indian mountaineer Tenzing Norgay successfullyclimbed the peak together. Since then, more than 7,200 individuals have made it to the top of Everestincluding more than 1,600 climbers who have completed multiple summits, according tostatistics compiled by Arnette.Get the latest stories in your inbox every weekday.Filed Under: History, Mountains, Outdoor Travel, Pollution, Tourism, Travel

Cerebras Systems launches DeepSeek's R1-70B AI model on its wafer-scale processor, delivering 57x faster speeds than GPU solutions and challenging Nvidia's AI chip dominance with U.S.-based inference processing.Read More

USC Games announced its 9th annual USC Games Expowill return to USCs campus on May 13 with more than 60 student games.Read More

Generative AI art made without human "creativity" cannot be protected by copyright lawsRepeatedly revising prompts does not change the US Copyright's Office's position, eitherImage credit: US Copyright Office News by Vikki Blake Contributor Published on Jan. 30, 2025 The US Copyright Office has confirmed that unless an author or artist contributes to the "creative process" of generative AI, it "cannot be protected by copyright" laws.The report by the US Copyright Office affirms that copyright protection in the United States "requires human authorship", giving the example that photographs taken by a monkey, for instance, "could not be protected".However, authors could "make the grade [to qualify for copyright] quite easily" as "the requisite level of creativity is extremely low"."In most cases, however, humans will be involved in the creation process, and the work will be copyrightable to the extent that their contributions qualify as authorship," the report explains."It is axiomatic that ideas or facts themselves are not protectible by copyright law, and the Supreme Court has made clear that originality is required, not just time and effort. In Feist Publications, Inc. v. Rural Telephone Service Co., the Court rejected the theory that 'sweat of the brow' alone could be sufficient for copyright protection."The vast majority of works make the grade quite easily, as they possess some creative spark, 'no matter how crude, humble or obvious' it might be."Repeatedly revising generative art prompts does not change the office's position, either."No matter how many times a prompt is revised and resubmitted, the final output reflects the users acceptance of the AI systems interpretation, rather than authorship of the expression it contains," the report stated.The team behind Catly, a pet sim announced at last month's The Game Awards, was forced to deny its debut trailer was created using generative AI after rumours began to swirl that the game may use blockchain tech or NFTs after some connected the studio's co-founder, Kevin Yeung, to other blockchain projects he is working on with a different team at TenthPlanet.

Jan 29Andrew WebsterDino Crisis launches on PC as GOG adds new tool to bring back more classicsImage: CapcomA survival horror classic is getting a new life on PC and it might be the first of many. PC games shop GOG announced the release of Dino Crisis and its sequel, which are both available starting today. Alongside the launch, the service also announced a new tool called dreamlist, with the goal of letting players vote on which old games that want to see hit the service next.For the uninitiated, Dino Crisis is essentially Resident Evil with dinosaurs. Thats not an insult: the first game was directed by original Resident Evil director Shinji Mikami, so the two series have a lot in common. Dino Crisis features the same methodical, survival horror gameplay but with faster (and occasionally much bigger) enemies to contend with. GOG says the new port supports Windows 10 / 11 and includes updates like support for ~4K resolution and modern gamepads. The two games can be purchased individually on GOG or as a bundle.Read Article >Nov 13, 2024Ash ParrishGOGs new preservation program intends to keep classic games playable foreverPC game platform GOG has launched a new preservation program dedicated to keeping beloved older games playable, now and in the future.If a game is part of the Preservation Program, it means that we commit our own resources to maintaining its compatibility with modern and future systems, the announcement blog reads.Read Article >Oct 25, 2024Sean HollisterVideo game preservationists have lost a legal fight to study games remotelyCollage by Cath Virginia / The Verge | Photos from Getty ImagesWhen video game scholars want to study games that are no longer on sale, they sometimes have to drive many hours to do it legally and that wont be changing anytime soon. The US Copyright Office has just denied a request from video game preservationists to let libraries, archives and museums temporarily lend individuals some virtual, remotely accessible copies of those works.Kendra Albert, who made the argument on behalf of the Software Preservation Network and the Library Copyright Alliance, says preservationists werent asking for a lot: It was the thing that basically exists for all kinds of special collections in libraries: the library reviews the request, makes sure its not harmful, and allows access to the work.Read Article >Sep 17, 2024Jay PetersSpellbreak lives.The game fantasy battle royale game was shut down, but a community version was released on Itch.io at the end of 2022 that has a thriving community around it, as reported by Aftermath. I wish more games would get second lives like this.Spellbreak, Shut Down When Its Developer Was Acquired By Blizzard, Is Still Alive - Aftermath[aftermath.site]Jun 26, 2024Andrew WebsterSurvival horror classics, sans DRM.Classic gaming site GOG.com is bringing the original Resident Evil series to its service, which means you can play the PlayStation-era horror trilogy on PC without fussing with DRM. The first game is available now, while parts two and three are on their way.Image: CapcomJun 25, 2024Andrew WebsterAn excellent retro collection gets even better.The playable documentary Atari 50 is getting an expansion that will add 39 classic games, split across two new interactive timelines, with even more behind-the-scenes details. Its not clear yet when the update will drop (or for how much), but later this year the collection is also getting a physical release.Mar 8, 2024Ash ParrishOne fan spent three years saving a Final Fantasy game before it shut downImage: Square EnixFebruary 29th was the last day of service for Dissidia Final Fantasy Opera Omnia, a mobile game based on Square Enixs Final Fantasy fighting game series. Since launching in the US in 2018, the game has amassed over 170 characters from across 30 years of Final Fantasy history, scattered throughout four acts composed of multiple chapters and side stories representing hundreds of hours of a game that can no longer be played. Though Opera Omnia is officially gone, one person has shouldered the task to ensure the game hasnt completely disappeared.On June 6th 2021 I began to work towards my goal of recording and rendering every Dissidia Final Fantasy Opera Omnia cutscene and upload[ing] them to YouTube, wrote Hatok, a video editor and self-described video game enthusiast. And now, 7 hours from the end of service, Ive reached the end of this massive project.Read Article >Jan 18, 2024Jay PetersAnother cool piece of Limited Run Games history.The companys physical version of Shantae and the Pirates Curse was was literally the last game produced on the US 3DS production line, with our cartridge order managing to get to Nintendo at the absolute last possible second, CEO Josh Fairhurst says on X. The game is Limited Runs final 3DS release.You can see the post on X. Screenshot by Jay Peters / The VergeJan 11, 2024Jay PetersFelix the Cat and Sparkster are back.Konami and Limited Run Games are partnering on physical rereleases of two Felix the Cat games and three Rocket Knight games for PS4, PS5, and Nintendo Switch. I was obsessed with Rocket Knight Adventures as a kid, so Im happy to see it available in physical formats on modern consoles.Image: Konami and Limited Run GamesDec 6, 2023Andrew WebsterJeff Minter is getting a playable documentary.Digital Eclipse is continuing its interactive documentary series with an upcoming collection about game designer and noted llama enthusiast Jeff Minter. The studio previously made Atari 50 and The Making of Karateka, and the new release will feature 42 different and very trippy classic games when it launches in 2024.Dec 5, 2023Jay PetersPlayStation keeps reminding us why digital ownership sucksPhoto by Vjeran Pavic / The VergeIn less than a week, Sony has given us two timely reminders of the tenuousness of digital ownership and both reminders involve things on PlayStation.Last week, Sony said that, because of content licensing arrangements, users wouldnt be able to watch Discovery content theyve purchased and that the content would be removed from their libraries as of December 31st, 2023. The resulting list of shows that will suddenly disappear because of corporate agreements is very long. Shows disappearing from streaming services is commonplace, but in this case, people are losing access to shows they bought to watch on demand whenever they wanted.Read Article >Dec 1, 2023Andrew WebsterAtari 50 is getting a big update.Now that developer Digital Eclipse is under the Atari umbrella, the studio is about to drop the first major update for its brilliant retro collection / playable documentary Atari 50. It launches on December 5th, and will add 12 new (old) Atari 2600 games. Unfortunately, its not clear yet what games will be coming, but it sounds like this is just the start of many updates to come.Nov 13, 2023David PierceThe fight to save old video gamesIllustration by Samar Haddad / The VergeWhen I ask Frank Cifaldi, the founder and director of the Video Game History Foundation, to explain the importance of preserving and maintaining old video games, he answers with a movie analogy. Imagine, he said, if movies were only released on, like, VHS, ever. You want to watch Back to the Future? All right, you have to go on eBay, and you have to find an antique VHS copy thats degraded a bit from use. You have to find a VCR that works, a TV that it plugs into or the external scalers that make it look correct on your modern TV and you might need a time-base corrector because the magnetic flux signal is out of sync.For too many games, this is the state of the industry. For the most part, decades worth of games now exist only in their original form: on a disk or cartridge that goes into a console nobody has anymore. Many of those games are going to be hard for players to ever find again and if we dont do anything to save them, they might disappear altogether.Read Article >Oct 31, 2023Jon PorterAtari is buying the developer behind its excellent 50th anniversary retro compilationImage: AtariAtari is acquiring Digital Eclipse, a studio that specializes in preserving and rereleasing retro games on modern hardware. The releases are often rich in additional materials and historical content. Think the Criterion Collection but for video games. In a press release, Atari says its paying up to $20 million for the studio, including an initial $6.5 million paid in a combination of cash and shares and a further $13.5 million, which is due to be paid in cash over the next decade subject to Digital Eclipses performance. It expects to complete the deal in the coming days.The two companies previously worked together on last years excellent Atari 50: The Anniversary Celebration, which included ports of over 90 classic games as well as unreleased prototypes and neat extras like short documentaries and old photos and magazine articles. But Digital Eclipse has also produced a number of other well-received rereleases of classic games including Teenage Mutant Ninja Turtles: The Cowabunga Collection and the interactive documentary The Making of Karateka.Read Article >Oct 16, 2023Jay PetersThe Portland Retro Gaming Expo helps keep the classics aliveSega had their own console?That was a devastating comment I heard as I sat playing Sonic the Hedgehog 2 on a Sega Genesis Mini at the Portland Retro Gaming Expo (PRGE) this weekend. The Sonic games on Genesis were my favorites as a kid; one time, I played so much Sonic that I peed my pants. How could this person not know about Sega consoles?Read Article >Oct 10, 2023Jay PetersAnother Stadia exclusive just ate a Power Pellet.Pac-Man Mega Tunnel Battle: Chomp Champs, a battle royale take on Pac-Man that was previously only available on Google Stadia before Stadia shut down, will get another chance on Nintendo Switch, PS5, PS4, Xbox Series X / X, Xbox One, and Steam, Bandai Namco announced on Tuesday. Its set to release in early 2024.Also, I regret to inform you that Nintendo just shut down Pac-Man 99. We knew the shutdown was coming, but Im still sad to see it go.Sep 28, 2023Andrew WebsterA fun way to support video game preservation.The Video Game History Foundation does great work when it comes to preserving the history of the medium, and now the institute is offering a unique way for people to show support. Right now on GOG you can purchase a supporter pack filled with all kinds of rare art and promotional materials for franchises like Yakuza and The Elder Scrolls, with proceeds going towards the VGHF.Check out this interview for more on the work theyre doing.GOG Interview: learn more about games preservation with Video Games History Foundation and support their cause![GOG.com]Sep 15, 2023Andrew WebsterThe original Wizardry has been remastered and you can play it right nowFresh off the launch of The Making of Karateka, retro game studio Digital Eclipse has announced a remaster of the original action RPG title Wizardry. Even better its out now in early access on both Steam and GOG.First released in 1981 and developed by Sir-Tech Software, the first Wizardry or Wizardry: Proving Grounds of the Mad Overlord, to give it its very long full title was a first-person, party-based, D&D-inspired RPG that proved incredibly influential and kicked off a long-running franchise. Heres the official description for the uninitiated:Read Article >Aug 29, 2023Andrew WebsterDigital Eclipse is preserving classic games in the most entertaining way possibleThe Making of Karateka. Image: Digital EclipseThe first game you play in The Making of Karateka, a release about the making of the cinematic action game Karateka, is an Asteroids clone. It may seem strange, but the attempt at recreating Asteroids was a seminal moment for Karateka creator Jordan Mechner, a humble beginning for a designer who would go on to create some of the most influential action games ever made, including Prince of Persia. The clone appears early on during a long, winding, and interactive timeline, which takes you through Mechners early days of experimentation before getting into how Karateka was developed and released.For studio Digital Eclipse, the team behind the interactive documentary, its another attempt at telling the story of classic games in a way thats completely native to the medium. As classic games become increasingly endangered, this approach is an important tool for keeping them alive. I think its the best way to tell these stories, says Digital Eclipse president Mike Mika. If youre going to tell the story of an interactive experience, it better be interactive.Read Article >Jul 14, 2023Ash ParrishAlmost 90 percent of classic games are critically endangered, say archivistsPhoto by Vjeran Pavic / The VergeWith advancing console generations and the slow demise of both backward compatibility and digital game storefronts, the ability to play older games has always been tough. Through a new study from the Video Game History Foundation (VGHF) and the Software Preservation Network, we now have a better understanding of just how difficult accessing older games really is.87% of classic video games released in the United States are critically endangered, wrote Kelsey Lewin, co-director of the Video Game History Foundation.Read Article >Dec 25, 2022Ash ParrishThe quest to save a Stadia exclusiveImage: Q-GamesEarlier this year, Google announced that it was shutting down its game streaming service Stadia, a short three years since its launch in 2018. While its mostly fans of the service feeling the impact of the closure, there are a handful of developers with Stadia exclusives that will unfortunately lose their games when the service shuts down for good in January. One of those is Q-Games, makers of PixelJunk Raiders. The Verge spoke with Q-Games founder and CEO, Dylan Cuthbert, who explained the unique situation Q-Games is in, trying to get their exclusive off Stadias foundering ship and somewhere safe where people can play it.PixelJunk Raiders is a space exploration roguelike that takes advantage of Stadias unique state share feature that allows people to share instances of their game in which other players can jump into and experience for themselves. Read Article >Nov 11, 2022Andrew WebsterAtari 50 is an incredible playable tour through video game historyAtari Karts on the Jaguar. Image: AtariOne of the biggest challenges in video game preservation is figuring out how to actually present old games. In 2022, there are more ways than ever to play the classics, whether its mini consoles, updated hardware, subscription services, retro collections, or modern rereleases. While these can make old games playable to new audiences, they arent always able to put them in a proper context which is especially important for really old games like, say, Adventure on the Atari 2600.But an expansive new release, made by Digital Eclipse to celebrate Ataris 50th anniversary, is the best attempt at a retro collection Ive ever experienced. Its available on just about every console out there right now as well as the PC, letting me put my PS5 to use for its intended purpose: playing Asteroids. The collection is huge, detailed, and does an amazing job of explaining why these games are so important.Read Article >Oct 12, 2022Ash ParrishTwo never-published NES games are up for auction on eBayPhoto by TENGKU BAHAR / AFP via Getty ImagesWhen Frank Cifaldi, founder and co-director at the Video Game History Foundation, finds an unreleased original NES game on eBay, its cause for celebration. Unearthing never-before-published games is his research kink, and according to him, such discoveries only happen once every five years. Yesterday, he found two.One game, Battlefields of Napoleon, got so far along in its development process that it has finalized box art, but the game never saw the light of a retail store. The other game is even rarer. Its a demo for the infamous Power Glove developed by Rare (yes, apparently that Rare) and is one of only five known games for the peripheral in existence (if you count the two games Nintendo announced but never released). Cifaldi is now working to raise funds to win both auctions so these never-before-seen pieces of video game history can be preserved and their contents shared with the public.Read Article >Sep 1, 2022Alicia HaddickThe ticking time bomb of modern free-to-play gamesNintendos Dragalia Lost is shutting down at the end of 2022. Photo by Amelia Holowaty Krales / The VergeDragalia Lost launched in 2018 as a statement of intent from Nintendo in partnership with Japanese developer Cygames. Nintendo may have first jumped into the field of mobile games in 2016 with the launch of games like Super Mario Run and Miitomo, but this was the first original property the company had produced exclusively for mobile devices. This free-to-play gacha game (a game whose content is generally free to access while charging microtransactions for loot boxes and randomized lotteries for rare and limited-time characters) had a flashy multi-region launch campaign collaborating with major Japanese musician DAOKO, banking on the games success at home and abroad.And it was a hit. Less than a year after launch, the game had already earned over $100 million, with a steady stream of merchandise following soon after. Yet, as of last month, Nintendo and Cygames published the games final update, and this week, it was revealed that the game would shut down on November 29th after just three years of operation.Read Article >Apr 19, 2022Ash ParrishHabitat for humanity: how a classic MMO got a second lifeIllustration by Melissa Mathieson / The VergeHabitat, the worlds first MMO developed for the Commodore 64 personal computer, went offline in 1992. It came back online in 2017 through the efforts of MADE, the Museum of Art and Digital Entertainment.Founded by Alex Handy in 2011, MADE seeks to legitimize the preservation of video games as both a historic and artistic medium within the context of our time. To that end, MADE amassed a collection of working video game consoles and a library of old games for patrons to play. Read Article >

DeepSeek has secured a completely open database that exposed user chat histories, API authentication keys, system logs, and other sensitive information, according to cloud security firm Wiz. The security researchers said they found the Chinese AI startups publicly accessible database in minutes, with no authentication required.The exposed information was housed within an open-source data management system called ClickHouse and consisted of more than 1 million log lines. As noted by Wiz, the exposure allowed for full database control and potentialprivilege escalation within the DeepSeek environment, which couldve given bad actors access to the startups internal systems. These findings were first reported by Wired.DeepSeek promptly secured the database after Wiz notified the startup about the issue.Its still not clear whether anyone else was able to access the exposed data, but the researchers told Wired, it wouldnt be surprising, given how simple it was to discover. Wizs researchers also told the outlet that DeepSeeks systems are designed similarly to those used by OpenAI, down to details like the format of the API keys. OpenAI accused DeepSeek of using its data to train its AI models earlier this week.

GameDev Archives GameDev ArchivesSelect Month January 2025 (17) December 2024 (29) November 2024 (25) October 2024 (26) September 2024 (22) August 2024 (14) July 2024 (20) June 2024 (20) May 2024 (21) April 2024 (23) March 2024 (30) February 2024 (23) January 2024 (17) December 2023 (18) November 2023 (17) October 2023 (30) September 2023 (30) August 2023 (13) July 2023 (17) June 2023 (20) May 2023 (24) April 2023 (20) March 2023 (25) February 2023 (22) January 2023 (19) December 2022 (15) November 2022 (25) October 2022 (18) September 2022 (19) August 2022 (18) July 2022 (13) June 2022 (22) May 2022 (23) April 2022 (28) March 2022 (29) February 2022 (21) January 2022 (21) December 2021 (18) November 2021 (26) October 2021 (22) September 2021 (21) August 2021 (23) July 2021 (24) June 2021 (23) May 2021 (24) April 2021 (28) March 2021 (35) February 2021 (26) January 2021 (28) December 2020 (33) November 2020 (38) October 2020 (34) September 2020 (38) August 2020 (30) July 2020 (33) June 2020 (34) May 2020 (27) April 2020 (34) March 2020 (31) February 2020 (26) January 2020 (22) December 2019 (20) November 2019 (23) October 2019 (34) September 2019 (30) August 2019 (17) July 2019 (25) June 2019 (22) May 2019 (26) April 2019 (21) March 2019 (28) February 2019 (24) January 2019 (27) December 2018 (14) November 2018 (19) October 2018 (22) September 2018 (23) August 2018 (23) July 2018 (30) June 2018 (32) May 2018 (33) April 2018 (21) March 2018 (29) February 2018 (16) January 2018 (18) December 2017 (15) November 2017 (15) October 2017 (15) September 2017 (20) August 2017 (12) July 2017 (15) June 2017 (25) May 2017 (20) April 2017 (16) March 2017 (30) February 2017 (31) January 2017 (24) December 2016 (32) November 2016 (42) October 2016 (27) September 2016 (34) August 2016 (32) July 2016 (35) June 2016 (37) May 2016 (48) April 2016 (83) March 2016 (56) February 2016 (43) January 2016 (54) December 2015 (44) November 2015 (49) October 2015 (17) September 2015 (13) August 2015 (13) July 2015 (17) June 2015 (27) May 2015 (20) April 2015 (15) March 2015 (18) February 2015 (9) January 2015 (11) December 2014 (17) November 2014 (22) October 2014 (47) September 2014 (9) August 2014 (5) July 2014 (14) June 2014 (17) May 2014 (12) April 2014 (13) March 2014 (18) February 2014 (13) January 2014 (15) December 2013 (6) November 2013 (8) October 2013 (8) September 2013 (21) August 2013 (13) July 2013 (16) June 2013 (17) May 2013 (13) April 2013 (21) March 2013 (18) February 2013 (13) January 2013 (13) December 2012 (15) November 2012 (22) October 2012 (24) September 2012 (24) August 2012 (41) July 2012 (16) June 2012 (32) May 2012 (28) April 2012 (19) March 2012 (12) February 2012 (18) January 2012 (11) December 2011 (12) November 2011 (13) October 2011 (22) September 2011 (7) August 2011 (11) July 2011 (5) June 2011 (7) May 2011 (16) April 2011 (10) January 1970 (1)

Tokenization plays a fundamental role in the performance and scalability of Large Language Models (LLMs). Despite being a critical component, its influence on model training and efficiency remains underexplored. While larger vocabularies can compress sequences and reduce computational costs, existing approaches tie input and output vocabularies together, creating trade-offs where scaling benefits larger models but harms smaller ones. This paper introduces a framework called Over-Tokenized Transformers that reimagines vocabulary design by decoupling input and output tokenization, unlocking new pathways for model efficiency and performance.Reference: https://arxiv.org/pdf/2501.16975Traditional tokenization methods use identical vocabularies for input processing and output prediction. While larger vocabularies allow models to process longer n-gram tokens (e.g., multi-character sequences), they force smaller models to handle overly granular output predictions, increasing underfitting risks. For instance, a 3-gram tokenizer reduces sequence length by 66% but requires predicting three characters jointlya task manageable for large models but overwhelming for smaller ones. Previous work like multi-token prediction (MTP) attempted to address this by predicting future tokens in parallel, but these methods still entangled input/output granularity and struggled with smaller architectures.The research team identified a critical insight through synthetic experiments with context-free grammars: input and output vocabularies influence models differently. Larger input vocabularies consistently improved all model sizes by enriching context representations through multi-gram embeddings. Conversely, larger output vocabularies introduced fine-grained prediction tasks that only benefited sufficiently large models. This dichotomy motivated their Over-Tokenized framework, which separates input encoding (Over-Encoding) and output decoding (Over-Decoding) vocabularies.Over-Encoding (OE) scales input vocabularies exponentially using hierarchical n-gram embeddings. Instead of a single token ID, each input token is represented as the sum of 1-, 2-, and 3-gram embeddings. For example, the word cat might decompose into embeddings for c, ca, and cat, allowing the model to capture multi-scale contextual cues. To avoid impractical memory costs from large n-gram tables (e.g., 100k entries), the team used parameter-efficient techniques:Modulo-based token hashing: Maps n-gram tokens to a fixed-size embedding table using modular arithmetic, enabling dynamic vocabulary expansion without storing all possible combinations.Embedding decomposition:Over-Decoding (OD) approximates larger output vocabularies by predicting multiple future tokens sequentially, a refinement of earlier MTP methods. For instance, instead of predicting one token at a time, OD trains the model to predict the next two tokens conditioned on the first prediction. Crucially, OD is selectively appliedonly larger models benefit from this granular supervision, while smaller ones retain single-token decoding to avoid underfitting.The researchers performed experiments on OLMo and OLMoE architectures and demonstrated three key findings:Log-Linear Scaling: Training loss decreased linearly as input vocabulary size grew exponentially (Figure 1). A 400M parameter model with a 12.8M-entry input vocabulary matched the performance of a 1B-parameter baseline, achieving 2.5 effective scaling at equal computational cost.Convergence Acceleration: Over-Encoding reduced training steps needed for convergence by 35 across tasks like MMLU and PIQA, suggesting richer input representations accelerate learning.Sparse Parameter Efficiency: Despite using 128 larger input vocabularies, memory and computation overheads increased by <5% due to sparse embedding access and optimized sharding strategies.On evaluations, the framework demonstrated consistent performance improvements across various model types. For dense models, a 151M Over-Encoded (OE) model achieved a 14% reduction in perplexity compared to its baseline. Similarly, in sparse Mixture-of-Experts (MoE) models, the OLMoE-1.3B with OE reduced validation loss by 0.12 points, although the gains were less pronounced as the benefits of sparse experts diluted the impact of embedding enhancements. Beyond synthetic experiments, real-world evaluations on large-scale datasets further validated these findings. Over-Encoded models consistently improved performance across multiple benchmarks, including MMLU-Var, Hellaswag, ARC-Challenge, ARC-Easy, and PIQA. Notably, the framework accelerated convergence, achieving a 5.7 speedup in training loss reduction. Additionally, downstream evaluations showed significant acceleration, with OE delivering speedups of 3.2 on MMLU-Var, 3.0 on Hellaswag, 2.6 on ARC-Challenge, 3.1 on ARC-Easy, and 3.9 on PIQA, highlighting its efficiency and effectiveness across diverse tasks.In conclusion, this work redefines tokenization as a scalable dimension in language model design. By decoupling input and output vocabularies, Over-Tokenized Transformers break traditional trade-offs, enabling smaller models to benefit from compressed input sequences without grappling with overly complex prediction tasks. The log-linear relationship between input vocabulary size and performance suggests embedding parameters represent a new axis for scaling laws, complementing existing work on model depth and width. Practically, the framework offers a low-cost upgrade path for existing architecturesintegrating Over-Encoding requires minimal code changes but yields immediate efficiency gains. Future research could explore hybrid tokenization strategies or dynamic vocabulary adaptation, further solidifying tokenizations role in the next generation of efficient, high-performing LLMs.Check out the Paper. All credit for this research goes to the researchers of this project. Also,dont forget to follow us onTwitter and join ourTelegram Channel andLinkedIn Group. Dont Forget to join our70k+ ML SubReddit.(Promoted) Vineet KumarVineet Kumar is a consulting intern at MarktechPost. He is currently pursuing his BS from the Indian Institute of Technology(IIT), Kanpur. He is a Machine Learning enthusiast. He is passionate about research and the latest advancements in Deep Learning, Computer Vision, and related fields.Vineet Kumarhttps://www.marktechpost.com/author/vineet1897/Creating An AI Agent-Based System with LangGraph: A Beginners GuideVineet Kumarhttps://www.marktechpost.com/author/vineet1897/Unlocking Autonomous Planning in LLMs: How AoT+ Overcomes Hallucinations and Cognitive LoadVineet Kumarhttps://www.marktechpost.com/author/vineet1897/Plurai Introduces IntellAgent: An Open-Source Multi-Agent Framework to Evaluate Complex Conversational AI SystemVineet Kumarhttps://www.marktechpost.com/author/vineet1897/Create Portrait Mode Effect with Segment Anything Model 2 (SAM2) [Recommended] Join Our Telegram Channel